Jump to content


Photo

SMF 2.0.X -> IPB, Members w/missing avatars


Origin topic

Converting from SMF v2.0.2 -> IPB v3.3.3, Members

Error Description:
When importing the members, if there are members with missing avatars (or invalid avatar URLs) a message is added to a log. For smaller boards this isn't a problem, but when importing thousands of users have this issue it creates a large log - large enough to overflow a MySQL text column.


mySQL query error: UPDATE core_sys_conf_settings SET conf_value='a:866:{i:0;s:68:\": Could not locate file ../path/to/avatars/iconalbert.gif\";i:1;s:79:\": Could not locate file ../path/to/avatars/c0cc6a023d5c2fb698dbf.jpg\";i:2;s:40:\"10: Could not fetch remote picture file.\";i:3;s:79:\": Could not locate file ../path/to/avatars/cbadc0193d62103b2afe8.jpg\";i:4;s:68:\": Could not locate file ../path/to/avatars/iconalbert.gif\";i:5;s:182:\"19: Contains one or more of these illegal characters: [ ] | ; ,  $ \ < > " with name [SG]TaSSaDaR. Member has still been created but with username as -SG-TaSSaDaR\";i:6;s:60:\": Could not locate file ../path/to/avatars/B2.jpg\";i:7;s:40:\"25: Could not fetch remote picture file.\";i:8;s:40:\"30: Could not fetch remote picture file.\";i:9;s:40:\"40: Could not fetch remote picture file.\";i:10;s:40:\"55: Could not fetch remote picture file.\";i:11;s:67:\": Could not locate file ../path/to/avatars/iconboy11.gif\";i:12;s:59:\": Could not locate file ../path/to/avatars/5.gif\";i:13;s:41:\"111: Could not fetch remote picture file.\";i:14;s:41:\"138: Could not fetch remote picture file.\";i:15;s:59:\": Could not locate file ../path/to/avatars/6.gif\";i:16;s:68:\": Could not locate file ../path/to/avatars/iconjakson.gif\";i:17;s:41:\"184: Could not fetch remote picture file.\";i:18;s:41:\"185: Could not fetch remote picture file.\";i:19;s:41:\"196: Could not fetch remote picture file.\";i:20;s:41:\"224: Could not fetch remote picture file.\";i:21;s:59:\": Could not locate file ../path/to/avatars/2.gif\";i:22;s:41:\"255: Could not fetch remote picture file.\";i:23;s:41:\"282: Could not fetch remote picture file.\";i:24;s:41:\"305: Could not fetch remote picture file.\";i:25;s:41:\"323: Could not fetch remote picture file.\";i:26;s:41:\"342: Could not fetch remote picture file.\";i:27;s:41:\"349: Could not fetch remote picture file.\";i:28;s:59:\": Could not locate file ../path/to/avatars/6.gif\";i:29;s:41:\"438: Could not fetch remote picture file.\";i:30;s:41:\"461: Could not fetch remote picture file.\";i:31;s:41:\"463: Could not fetch remote picture file.\";i:32;s:59:\": Could not locate file ../path/to/avatars/6.gif\";i:33;s:59:\": Could not locate file ../path/to/avatars/1.gif\";i:34;s:41:\"512: Could not fetch remote picture file.\"; [...]}' WHERE conf_id=306
SQL error: Data too long for column 'conf_value' at row 1
SQL error code: 1406
Date: Friday 06th July 2012 11:47:22 PM


 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 Date: Fri, 06 Jul 2012 23:47:22 +0000
 Error: 1406 - Data too long for column 'conf_value' at row 1
 IP Address: 127.0.0.1 - /admin/index.php?adsess=a69d30609103c104bb953f96106d1a73&app=convert&app=convert&module=board&section=smf&do=members&st=13000&cycle=1000&total=130767
 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
 mySQL query error: UPDATE core_sys_conf_settings SET conf_value='a:866:{i:0;s:68:\": Could not locate file ../path/to/avatars/iconalbert.gif\";i:1;s:79:\": Could not locate file ../path/to/avatars/c0cc6a023d5c2fb698dbf.jpg\";i:2;s:40:\"10: Could not fetch remote picture file.\";i:3;s:79:\": Could not locate file ../path/to/avatars/cbadc0193d62103b2afe8.jpg\";i:4;s:68:\": Could not locate file ../path/to/avatars/iconalbert.gif\";i:5;s:182:\"19: Contains one or more of these illegal characters: [ ] | ; ,  $ \ < > " with name [SG]TaSSaDaR. Member has still been created but with username as -SG-TaSSaDaR\";i:6;s:60:\": Could not locate file ../path/to/avatars/B2.jpg\";i:7;s:40:\"25: Could not fetch remote picture file.\";i:8;s:40:\"30: Could not fetch remote picture file.\";i:9;s:40:\"40: Could not fetch remote picture file.\";i:10;s:40:\"55: Could not fetch remote picture file.\";i:11;s:67:\": Could not locate file ../path/to/avatars/iconboy11.gif\";i:12;s:59:\": Could not locate file ../path/to/avatars/5.gif\";i:13;s:41:\"111: Could not fetch remote picture file.\";i:14;s:41:\"138: Could not fetch remote picture file.\";i:15;s:59:\": Could not locate file ../path/to/avatars/6.gif\";i:16;s:68:\": Could not locate file ../path/to/avatars/iconjakson.gif\";i:17;s:41:\"184: Could not fetch remote picture file.\";i:18;s:41:\"185: Could not fetch remote picture file.\";i:19;s:41:\"196: Could not fetch remote picture file.\";i:20;s:41:\"224: Could not fetch remote picture file.\";i:21;s:59:\": Could not locate file ../path/to/avatars/2.gif\";i:22;s:41:\"255: Could not fetch remote picture file.\";i:23;s:41:\"282: Could not fetch remote picture file.\";i:24;s:41:\"305: Could not fetch remote picture file.\";i:25;s:41:\"323: Could not fetch remote picture file.\";i:26;s:41:\"342: Could not fetch remote picture file.\";i:27;s:41:\"349: Could not fetch remote picture file.\";i:28;s:59:\": Could not locate file ../path/to/avatars/6.gif\";i:29;s:41:\"438: Could not fetch remote picture file.\";i:30;s:41:\"461: Could not fetch remote picture file.\";i:31;s:41:\"463: Could not fetch remote picture file.\";i:32;s:59:\": Could not locate file ../path/to/avatars/6.gif\";i:33;s:59:\": Could not locate file ../path/to/avatars/1.gif\";i:34;s:41:\"512: Could not fetch remote picture file.\"; [...]}' WHERE conf_id=306
 .--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------.
 | File                                                                       | Function                                                                      | Line No.          |
 |----------------------------------------------------------------------------+-------------------------------------------------------------------------------+-------------------|
 | ../admin/sources/base/core.php                                             | [db_main_mysql].update                                                        | 1603              |
 '----------------------------------------------------------------------------+-------------------------------------------------------------------------------+-------------------'
 | ../admin/applications_addon/ips/convert/sources/interface_acp.php          | [IPSLib].updateSettings                                                       | 163               |
 '----------------------------------------------------------------------------+-------------------------------------------------------------------------------+-------------------'
 | ../admin/applications_addon/ips/convert/sources/lib_master.php             | [_interface].next                                                             | 2406              |
 '----------------------------------------------------------------------------+-------------------------------------------------------------------------------+-------------------'
 | ../admin/applications_addon/ips/convert/modules_admin/board/smf.php        | [lib_master].next                                                             | 542               |
 '----------------------------------------------------------------------------+-------------------------------------------------------------------------------+-------------------'
 |                                                                            | [admin_convert_board_smf].convert_members                                     |                   |
 '----------------------------------------------------------------------------+-------------------------------------------------------------------------------+-------------------'
 | ../admin/applications_addon/ips/convert/modules_admin/board/smf.php        | [].call_user_func                                                             | 90                |
 '----------------------------------------------------------------------------+-------------------------------------------------------------------------------+-------------------'
 | ../admin/sources/base/ipsController.php                                    | [admin_convert_board_smf].doExecute                                           | 306               |
 '----------------------------------------------------------------------------+-------------------------------------------------------------------------------+-------------------'

I've truncated the insert query for the above, but it is pretty long (67,533 chars - over the 65,535 limit)

Replication Steps:
  • Fresh install IPB
  • Install Converter
  • Import SMF2 data (if not already)
  • Have missing avatars, etc. from the users
  • New Converter for SMF2
  • Run "Members" Conversion
  • Error should occur here, for me after ~750 members

Possible Fixes:
This could be negated a bit by cleaning up the data before importing it. Right now, it's GIGO so-to-speak but IMO IPB should be able to safely handle this.
  • Remove DB logging altogether (keeping the filesystem logging is OK). For me, commenting out the following lines takes care of the issue:
    /admin/applications_addon/ips/convert/sources/lib_master.php Line 944, 1875
    $this->logError($id, 'Could not locate file '.$source.'/'.$fileloc);

    /admin/applications_addon/ips/convert/sources/lib_master.php Line 1458
    $this->logError ( $info['id'], 'Could not fetch remote picture file.' );

    /admin/applications_addon/ips/convert/sources/lib_downloads.php Line 693
    $this->logError ( $info['id'], 'Could not fetch remote picture file.' );

    /admin/applications_addon/ips/convert/sources/lib_gallery.php Line 426
    $this->logError($id, 'Could not locate file '.$source.'/'.$fileloc);

    /admin/applications_addon/ips/convert/sources/convertGallery.php Line 519
    $this->logError($id, 'Could not locate file '.$source.'/'.$fileloc);
  • Give an option for DB logging (personally I don't see the need to log to the DB if you're logging to a flat file and pulling from that to begin with)
  • Add a check to the length of the updateSettings function, and truncate / move to a new row as needed.
  • Log errors separately, in a different table (similar to the mail_error_log table)

System Details:
Win 7x64
Apache 2.4.2
PHP 5.3.14
MySQL 5.5.21

Status: Cannot Reproduce
Version: 1.2.3
Fixed In: 0


3 Comments

Photo
MisterPhilip
Jul 06 2012 07:12 PM
On a sidenote.. what's up with the spaces within the pre tags in the tracker? With monospace font the trace should be all lined up (with the correct browser width)
Updating Fixed In to: 0
Updating Status to: Cannot Reproduce
Updating Version to: 0

I'm confused here, the SQL error suggests that the converter is trying to insert things into a settings table somewhere but I don't see this from the code. It's possible that this has been fixed.

 

If you know that this still an issue can you please open a ticket.

Updating Version to: 1.2.3

-