phpBB32024-03-28T17:39:17Zhttps://d.moonfire.us/tags/phpbb3/D. MoonfireCreative Commons Attribution-NonCommercial-ShareAlike 4.0 InternationalUsing the PHPBB3 crawler convertor2013-11-12T06:00:00Zhttps://d.moonfire.us/blog/2013/11/12/using-the-phpbb3-crawler-convertor/<p>Last night, someone on Twitter asked about using a crawling convertor for a forum. In specific, a PHPBB3 to PHPBB3 conversion. However, the program involved was Python and a bit obscure to use. So, I offered to come up with a (possibly) easier set of directions of use.</p>
<p><!--more--></p>
<h1>What is needed</h1>
<p>There are a number of programs that need to be used to use this conversion.</p>
<ul>
<li><a href="http://www.python.org/getit/">Python 2.7.6</a>: Download the 2.7.6 (not 3.3.1) for Windows MSI installer for 32 or 64 bit, whatever is appropriate for your machine.</li>
<li>The bulk of the copying is done by the Converters scripts found <a href="https://www.phpbb.com/community/viewtopic.php?f=65&t=1761395">here</a>. This uses a single user's login to walk through the contents of a forum and generates the SQL files which can then be imported into another forum.</li>
</ul>
<h1>Setting up the user</h1>
<p>Before the program is run, you need to set up the user that will be walking the forum. Whatever this user sees will be converted, so if you have a private board or category, this user needs to have permissions for it.</p>
<p>Because this walks a forum, you need to make sure your forum is generated in a consistent manner. Missing these steps is a good way of having a lot of trouble.</p>
<ul>
<li>Login as the user you selected</li>
<li>Click on "User Control Panel"</li>
<li>Click on "Board Preferences"</li>
<li>Click on "Edit global settings"</li>
<li>Change language to "American English"</li>
<li>Change your board style to "prosilver"</li>
<li>Change your date format to Custom and enter "U" into the box.</li>
<li>Click "Submit"</li>
</ul>
<h1>Install Python</h1>
<p>This should be as simple as double-clicking the MSI and following the prompts.</p>
<h1>Installing and configuring the Converter</h1>
<ul>
<li>Download the Converters.zip file</li>
<li>Extract it to some location (<em>C:\TEMP\Converters</em> is what I'm going to use)</li>
<li>Go into that folder</li>
<li>Copy <em>common.py</em> and paste it into the <em>phpBB3</em> folder</li>
<li>Edit <em>settings.py</em>
<ul>
<li>Change table_prefix to match what you use (you can get this from the <em>config.php</em> file as <em>$table_prefix</em></li>
<li>Change URL to the site of your old forum (same thing you gave me in the message)</li>
<li>Change NEWURL to the new forum</li>
<li>Change CONVERT_ATTACHMENTS to "True" or "False"</li>
<li>Change RANDOMNUMBER to a big random number, important to avoid people from guessing your password</li>
<li>Change NUMUSERS to the highest user ID (or just big a really big number)</li>
<li>Save the file and exit</li>
</ul>
</li>
</ul>
<h1>Get the cookie</h1>
<p>This step basically logs into the site and retrieves the needed information to walk the site. On Windows, double-click the <code>phpbb3_get_cookie.py</code> script. It should open a window with some prompts, enter each one. As before, I'm using <em>Bob</em> as the login.</p>
<pre>Username? Bob
Password (hidden)?
Success: got cookie for user Bob
Press <enter> to quit.</enter></pre>
<p>Once this has run, there should be a file <em>cookie.txt</em> in the direction. There will also be a <em>critical_errors.log</em> which is hopefully empty.</p>
<h1>Start the walking</h1>
<p>This is the big step. Double-click on <em>members_topics_posts_phpBB3.py</em>:</p>
<pre>Downloading Members...
Member 1 ... Downloading -
HTTP Error ignored: 404 Not Found
Member 1 ... Warning: Member doesn't exist!
Member 2 ... Completed! User Password: 317887274831</pre>
<p>And a bit later...</p>
<pre>Forum index ... Completed
Category 1 ... Completed
Forum 2 ... Completed: 8 topics found
Forum 7 ... Completed: 153 topics found
Forum 10 ... Completed: 11 topics found
Category 44 ... Completed
Forum 12 ... Completed: 4 topics found
Forum 47 ... Completed: 4 topics found
Forum 48 ... Completed: 30 topics found</pre>
<p>And later...</p>
<pre>Topic ID 42 ... Completed: 3 posts
Topic ID 43 ... Completed: 1 post
Topic ID 44 ... Completed: 1 post
Topic ID 45 ... Completed: 3 posts</pre>
<p>Finally after a long time...</p>
<pre>Topic ID 1901 ... Completed: 4 posts
Elapsed Conversion Time 746.45 seconds
Converted 7469 posts and 1646 topics.
Complete! Press <enter> to exit.</enter></pre>
<h1>Importing the SQL</h1>
<p>The output of the above program will be three files:</p>
<ul>
<li>members_phpBB3.sql</li>
<li>categories_forums_phpBB3.sql</li>
<li>topics_posts_phpBB3.sql</li>
</ul>
<p><em>Note, running these scripts will pretty much blow away anything you have, so make sure you don't have anything in the forum first.</em></p>
<ul>
<li>Connect to your MySQL instance of your new forum</li>
<li>Run <em>members_phpBB3.sql</em> with utf8 encoding</li>
<li>Run <em>categories_forums_phpBB3.sql</em> with utf8 encoding</li>
<li>Run <em>topics_posts_phpBB3.sql</em> with utf8 encoding (if you haven't figured that out yet)</li>
</ul>
<p>Importing SQL is fairly easy. There are multiple ways, but I usually do this from the command line. Your ISP may provide a front-end for MySQL that does this.</p>
<ul>
<li>SSH into your ISP for the new forum</li>
<li>Run: mysql -u<em>username</em> -p --default-character-set=utf8 <em>database</em>
<ul>
<li><em>username</em> and <em>database</em> can be found from the config.php of the new forum.</li>
</ul>
</li>
<li>Add the "mysql>" prompt, type:
<ul>
<li>"SOURCE members_phpBB3.sql"</li>
</ul>
</li>
<li>Then repeat for the rest of the files.</li>
</ul>
<h1>Cleaning up the mess</h1>
<ul>
<li>Log into your new forum</li>
<li>Switch over to the Administrator Control Panel (ACP)</li>
<li>Click on the "Run now" next to Resynchronize statistics</li>
<li>Click on the "Run now" next to Resynchronize post counts</li>
<li>Click on the "Run now" next to Resynchronize dotted topics</li>
<li>Click on "Maintenance" tab</li>
<li>Click on "Search Index"</li>
<li>If there, click on "Delete index"</li>
<li>Click on "Create index"</li>
<li>Click on "Forums" tab</li>
<li>Go through every single forum, sub forum, and sub-sub forum and click on the yellow recycle icon (resynchronize).</li>
</ul>
<h1>Sending a changed password email</h1>
<ul>
<li>Review the settings "msgtitle" and "msgbody" in the <em>settings.py</em> file</li>
<li>Double-click <em>pm_passwords.py</em> to send out an email to all the users with the password
<ul>
<li>Alternatively, they can use "I forgot my password" and it should work</li>
</ul>
</li>
</ul>
<h1>Warning</h1>
<p>Now, these are fairly generic directions and terrible things can happen if I missed something, but they should be easier directions to follow than the easier link.</p>