Using the PHPBB3 crawler convertor

Last night, someone on Twitter asked about using a crawling convertor for a forum. In specific, a PHPBB3 to PHPBB3 conversion. However, the program involved was Python and a bit obscure to use. So, I offered to come up with a (possibly) easier set of directions of use.

What is needed

There are a number of programs that need to be used to use this conversion.

  • Python 2.7.6: Download the 2.7.6 (not 3.3.1) for Windows MSI installer for 32 or 64 bit, whatever is appropriate for your machine.
  • The bulk of the copying is done by the Converters scripts found here. This uses a single user's login to walk through the contents of a forum and generates the SQL files which can then be imported into another forum.

Setting up the user

Before the program is run, you need to set up the user that will be walking the forum. Whatever this user sees will be converted, so if you have a private board or category, this user needs to have permissions for it.

Because this walks a forum, you need to make sure your forum is generated in a consistent manner. Missing these steps is a good way of having a lot of trouble.

  • Login as the user you selected
  • Click on "User Control Panel"
  • Click on "Board Preferences"
  • Click on "Edit global settings"
  • Change language to "American English"
  • Change your board style to "prosilver"
  • Change your date format to Custom and enter "U" into the box.
  • Click "Submit"

Install Python

This should be as simple as double-clicking the MSI and following the prompts.

Installing and configuring the Converter

  • Download the Converters.zip file
  • Extract it to some location (C:\TEMP\Converters is what I'm going to use)
  • Go into that folder
  • Copy common.py and paste it into the phpBB3 folder
  • Edit settings.py
    • Change table_prefix to match what you use (you can get this from the config.php file as $table_prefix
    • Change URL to the site of your old forum (same thing you gave me in the message)
    • Change NEWURL to the new forum
    • Change CONVERT_ATTACHMENTS to "True" or "False"
    • Change RANDOMNUMBER to a big random number, important to avoid people from guessing your password
    • Change NUMUSERS to the highest user ID (or just big a really big number)
    • Save the file and exit

Get the cookie

This step basically logs into the site and retrieves the needed information to walk the site. On Windows, double-click the phpbb3_get_cookie.py script. It should open a window with some prompts, enter each one. As before, I'm using Bob as the login.

Username? Bob
Password (hidden)? 
Success: got cookie for user Bob
Press  to quit.

Once this has run, there should be a file cookie.txt in the direction. There will also be a critical_errors.log which is hopefully empty.

Start the walking

This is the big step. Double-click on members_topics_posts_phpBB3.py:

Downloading Members...
Member 1 ... Downloading -                                                     
HTTP Error ignored: 404 Not Found
Member 1 ... Warning: Member doesn't exist!                                    
Member 2 ... Completed! User Password: 317887274831

And a bit later...

Forum index ... Completed                                                      
Category 1 ... Completed                                                      
 Forum 2 ... Completed: 8 topics found                                        
 Forum 7 ... Completed: 153 topics found                                      
 Forum 10 ... Completed: 11 topics found                                      
Category 44 ... Completed                                                     
 Forum 12 ... Completed: 4 topics found                                       
 Forum 47 ... Completed: 4 topics found                                       
  Forum 48 ... Completed: 30 topics found

And later...

Topic ID 42 ... Completed: 3 posts                                             
Topic ID 43 ... Completed: 1 post                                              
Topic ID 44 ... Completed: 1 post                                              
Topic ID 45 ... Completed: 3 posts

Finally after a long time...

Topic ID 1901 ... Completed: 4 posts                                           
Elapsed Conversion Time 746.45 seconds
Converted 7469 posts and 1646 topics.
Complete! Press  to exit.

Importing the SQL

The output of the above program will be three files:

  • members_phpBB3.sql
  • categories_forums_phpBB3.sql
  • topics_posts_phpBB3.sql

Note, running these scripts will pretty much blow away anything you have, so make sure you don't have anything in the forum first.

  • Connect to your MySQL instance of your new forum
  • Run members_phpBB3.sql with utf8 encoding
  • Run categories_forums_phpBB3.sql with utf8 encoding
  • Run topics_posts_phpBB3.sql with utf8 encoding (if you haven't figured that out yet)

Importing SQL is fairly easy. There are multiple ways, but I usually do this from the command line. Your ISP may provide a front-end for MySQL that does this.

  • SSH into your ISP for the new forum
  • Run: mysql -uusername -p --default-character-set=utf8 database
    • username and database can be found from the config.php of the new forum.
  • Add the "mysql>" prompt, type:
    • "SOURCE members_phpBB3.sql"
  • Then repeat for the rest of the files.

Cleaning up the mess

  • Log into your new forum
  • Switch over to the Administrator Control Panel (ACP)
  • Click on the "Run now" next to Resynchronize statistics
  • Click on the "Run now" next to Resynchronize post counts
  • Click on the "Run now" next to Resynchronize dotted topics
  • Click on "Maintenance" tab
  • Click on "Search Index"
  • If there, click on "Delete index"
  • Click on "Create index"
  • Click on "Forums" tab
  • Go through every single forum, sub forum, and sub-sub forum and click on the yellow recycle icon (resynchronize).

Sending a changed password email

  • Review the settings "msgtitle" and "msgbody" in the settings.py file
  • Double-click pm_passwords.py to send out an email to all the users with the password
    • Alternatively, they can use "I forgot my password" and it should work

Warning

Now, these are fairly generic directions and terrible things can happen if I missed something, but they should be easier directions to follow than the easier link.

Metadata

Categories:

Tags: