Articles

Hello fellow Magentoers :)

We at OYE decided to make a test migration project to get a feel of what migration from magento1 to magento2 is like. This article only describes data migration, porting of custom functionality and theme will be described later, when we complete it :) The magento2 docs page describes the data migration as a straight forward process, let's see if it is true.

The first problem was to understand if the migration tool supports moving custom table columns from m1 to m2. After some research, it turned out that it can’t do that. So if your sales_flat_order table has some custom column where you store some data – it can’t be transferred to m2 with the default migration tool functionality. I think this feature will be widely needed and it will appear in the migration tool soon.

Let me guide you through the migration process. First you should setup a clean magento2 instance for data migration. After that is done, you can install the data migration tool as described in this doc http://devdocs.magento.com/guides/v2.0/migration/migration-tool-install.html

After that is done, open the file
vendor/magento/data-migration-tool/etc/ce-to-ce/1.9.1.0/config.xml.dist
copy it into config.xml and start editing.
First you should add the source and destination database mysql credentials:

    <source>
        <database host="localhost" name="sourceDb" user="root" password="root"/>
    </source>
    <destination>
        <database host="localhost" name="destinationDb" user="root" password="root"/>
    </destination>

Then you can launch the migration data tool from shell like this:
cd /%magento2path%/bin
php ./magento migrate:data -r config_file_absolute_path_here

The first time you launch it you will see a lot of errors like "Source fields not mapped", "Destination fields not mapped.", "Source documents not mapped" etc.
Referred as "document" here is a database table, a "field" is a column.
You should ignore all the fields that are listed in "Source fields not mapped" errors. For example lets take this error:
[ERROR]: Source fields not mapped. Document: sales_flat_order_address. Fields: giftregistry_item_id

To solve this you will have to copy the map.xml.dist into map.xml file, then change this line in vendor/magento/data-migration-tool/etc/ce-to-ce/1.9.1.0/config.xml

<map_file>etc/ce-to-ce/1.9.1.0/map.xml.dist</map_file>

like this

<map_file>etc/ce-to-ce/1.9.1.0/map.xml</map_file>

Do this with every map file you want to change - this way you will have default untouched files as .dist files.
Open the map.xml and add some xml to the section

<source>
  <field_rules>

the xml you need to add:

<ignore>
  <field>sales_flat_order_address.giftregistry_item_id</field>
</ignore>

Congrats. You solved the error "Source fields not mapped. Document: sales_flat_order_address. Fields: giftregistry_item_id"
For the error "Destination fields not mapped.." you would need to add new rows to this section

<destination>
  <field_rules>

Sometimes you will have tables on source database that don't exist in magento2.. then you will have add this code

<ignore>
   <document>adjholiday</document>
</ignore>

to this section

<source>
  <document_rules>

You will have to do this with every field you are getting an error for. And note that these errors come from different steps. The steps are

• EAV Step
• Customer Attributes Step
• Map Step
• Url Rewrite Step
• Log Step
• Ratings Step
• ConfigurablePrices step
• OrderGrids Step

For the map step - the corresponding config file is etc/ce-to-ce/1.9.1.0/map.xml
For the eav step - the file is <eav_map_file>etc/ce-to-ce/map-eav.xml
You can find the links to these files in
vendor/magento/data-migration-tool/etc/ce-to-ce/1.9.1.0/config.xml
So add

<ignore>

statements to corresponding files.

There are quite a lot of fields to be ignored, so I coded a small utility script to automate this. It generates the necessary ignore statements that you can insert into the xml configs.
Here's the code:

<h1>Ignore fields per table</h1>
<form method="post">
    <label>Fields</label>
    <input type="text" name="fields">
    <label>Table</label>
    <input type="text" name="table">
    <input type="submit">
</form>

<?php

if(isset($_POST['fields']))
{
    $fields = explode(',', $_POST['fields']);
    echo '<textarea rows="50" cols="100">';
    foreach ($fields as $field)
    {
        echo "<ignore>\r\n";
        echo sprintf("<field>%s.%s</field>\r\n", $_POST['table'], $field);
        echo "</ignore>\r\n";
    }
    echo '</textarea>';
}

?>

<h1>Ignore whole tables</h1>
<form method="post">
    <label>Tables, comma separated</label>
    <input type="text" name="tables2">
    <input type="submit">
</form>

<?php

if(isset($_POST['tables2']))
{
    $tables = explode(',', $_POST['tables2']);
    echo '<textarea rows="50" cols="100">';
    foreach ($tables as $table)
    {
        echo "<ignore>\r\n";
        echo sprintf("<document>%s</document>\r\n", $table);
        echo "</ignore>\r\n";
    }
    echo '</textarea>';
}

?>

The top form is for unmapped fields (columns) and the second form is for unmapped documents (tables). The script will generate pieces of xml that you should add to your config files.

Using the magento 2.0.0 and data-migration-tool 2.0.0 these are the things that i had to do to get the migration tool actually migrate something (I had to xdebug the console script to understand what's happening):

• add all custom and "not-mapped" columns to source/fields/ignore in map.xml
• add destination "not-mapped" columns to destination/fields/ignore in map.xml

Then I had to do some code changes in the migration tool to get it working:

-                'INSERT INTO %s (%s) VALUES %s',
+                'INSERT IGNORE INTO %s (%s) VALUES %s',

src/Migration/Resource/Adapter/Mysql.php file in method insertMultiple() as it was trying to insert some fields twice and threw an exception on that

The second change was this one

src/Migration/Step/UrlRewrite/Version191to2000.php

-        $result &= array_keys($this->source->getStructure(self::SOURCE)->getFie
-            == $this->structure[MapInterface::TYPE_SOURCE][self::SOURCE];
+        $a = array_keys($this->source->getStructure(self::SOURCE)->getFields())
+        $b = $this->structure[MapInterface::TYPE_SOURCE][self::SOURCE];
+        $result &= sort($a) == sort($b);

In the method integrity()
The problem is that the url_rewrite table columns order differ from the order in the corresponding xml config. I changed it so the order doesn't matter and the integrity check suceeds.

I've had another problem when I transferred data from upgraded magento 1.8 installation. Some data have not been fully transferred.

There was a value missing in row with entity type code = "catalog_product" from table eav_entity_type - this row had additional_attribute_table column = NULL. I had to set it to catalog_eav_attribute to get the product grid working. Do this if you get an exception on the product grid after migration.

One more problem i hard is that url rewrite step could not finish, because of different qty of rows in source and destination tables.. Source had 25771 and destination had 25770 rows, not sure why :) The volume check happens in this file:
vendor/magento/data-migration-tool/src/Migration/Step/UrlRewrite/Version191to2000.php::volume()

You can hack your way through this problem if you have it :)

I would suggest you use xdebug to debug the migration script. Check out this link https://confluence.jetbrains.com/display/PhpStorm/Debugging+PHP+CLI+scripts+with+PhpStorm
Launch the magento CLI by PHPSTORM with xdebug and you'll be able to discover cause of the problems faster.

When you want to start over the migration process (you screwed up the database on the magento2, data integrity is broken, basically this happens every time migration doesn't finish) - you need to do an uninstall of magento2 like this
/%magento2path%/bin/magento setup:uninstall

This will clear the database. Then install a clean magento2 db again like this:

/%magento2path%/bin/magento setup:install --backend-frontname=admin --db-name=dbname --db-user=username--db-password=password --base-url=http://localurl --admin-user=admin --admin-password=111111ab --admin-email=admin@email.com

I have this string written down because I need to use it quite often.

When you make a brand new attempt of migration, use --reset flag like this:
php migrate data --config ../etc/ce-to-ce/1.9.1.0/config.xml --reset

Otherwise, the tool will skip the steps that were completed before (but you will not know about it from console output). It really made me scratch my head for a while.

In the end, I could migrate magento 1.9.1 sample data, and I successfully migrated real project data from 1.8.1 (upgraded to 1.9.1)

I hope my case study will help you in your migration efforts :)
Please leave your feedback and share experience in the comments!

Cheers guys!