How to create Feeds
This is an advanced tutorial, that assumes at least some familiarity with Drupal content types, fields, and views.
Things Feeds are helpful for
- Creating news feeds, so that content with the appropriate tag can be imported from Western Today onto a department site
- Moving content from a Drupal 7 site to a Drupal 8 site, especially for content that is from types other than basic page
- Keeping content synced between two different Drupal sites (whether they on different versions, are both on 7, or are both on 8)
The example in this tutorial is one that was created for scenario 2: Moving content from a Drupal 7 site to a Drupal 8 site.
Overview of the process
I'll refer to the site you're exporting content from as the export site, and the site you're importing to the import site.
In general, how feeds works is that you:
- Create a view on the export site in some kind of RESTful data format (XML or JSON)
- Use the feeds module on the import site to to map the fields from that view into nodes on the import site
What to do on the export site
This tutorial's example uses JSON as the RESTful format, but you can use XML if you prefer working with that data format.
To create a view, you'll need to both have login access to the export site and appropriate permissions. If you do not have access to the site (for example, if you are attempting to create a Western Today news feed) you'll need to request help from WebTech in getting the view set up. You can also request help with getting permissions configured if you do have access to the export site, but are unable to create the view.
Enable Views modules
If the export site is a Drupal 7 site, you'll need to enable the module Views JSON or XML Views. If the module you need is not installed on the site, contact WebTech.
If the export site is a Drupal 8 site, enable the module RESTful Web Services
Create the view
Create a view of the content type that you are exporting.
- The view should be of the content type you are exporting.
- Make the view a page, and be sure to set the path
- Set the Display Format to JSON data document (or, XML data document)
- Eventually you will want to set the view to display all items, but during the initial setup and testing process, it's helpful to keep the number of items around 10. Do not use a pager.
Note for XML Views
Under the settings link next to XML Data Document, change the Root Element Name from node to nodes (plural). It defaults to both being node, which can cause errors later on in the process.
Filters are outside of the scope of this tutorial, so we won't be using them in the example here, but it's helpful to know they are available. You can use filters on this view if you only need to export some of the content of a given type from a site.
Add all of the fields that you wish to export. I find it helpful to create custom labels for the fields, with no special characters or spaces.It's helpful to open Structure > Content Types > Your content type > Manage Fields in a separate window to see the full list of fields that exist on the content type.
You may not need to export all of these fields. For example:
- For things like news feeds, you may not fields like the body text of the article. Instead you might just need things like the title and a url to link back to the full article.
- If you are cleaning house, this is a great opportunity to drop any fields that aren't being actively used anymore.
- There are also often system created fields that do not need to be exported.
You may need to do some advanced formatting in views to get your data in the appropriate format. Time and date fields especially generally need some extra attention. Things like link fields may need to be included twice, once for the link text and once for the link url, depending on what format you're using and your own personal preference for how you keep the data straight.
Final view example
What to do on the import site
Recreate the content type
On the import site, create the the content type you'd like to import the data into. For example, if this is news feed, create a "News" content type. If you are migrating content, the content type will likely be the same as it was on the export site.
The fields you will need to add on the new content type should match the fields on your export view, and be the appropriate field type to hold that data.
Enable Feeds modules
Enable Feeds, Feeds Extensible Parser, and Feeds Tamper (as well as any modules that they may require, such as Tamper)
Create a feed type
Structure > Feed Type > Add Feed Type
- Set the Fetcher to Download from url
- Parser should be JsonPath (or XML)
- Processor is usually Node (Though, you can import other entities, like users! See the special note at the end for User Feeds)
- Set the content type to whatever you created
- Set the import period to whatever makes sense. For things like News and Events feeds, you'll want this to be somewhere in the 15min - 1hr range, as those updates should come through often. For things like content migrations, I usually set this to Never so that I'm only ever doing manual imports.
Under Processor Settings:
- Check Update existing contents
- You may also want to check Force update under advance settings.
Then click the Save and add Mappings button.
Mappings are where we tell the feed how to map things from the XML file into the content on the import site. To set up a mapping:
- Set the Context to nodes[*].node for JSON feeds from Drupal 7, $.* for JSON feeds from Drupal 9, or //node for XML feeds
- Select a target, ie a field on the content type
- A Source drop down will then appear. Select New xpath source
- A text box will appear. This is where we'll put the field names from the JSON or XML document.
You'll do this for all of your fields. However, when creating a new feed, your best bet is to just get one or two fields mapped at a time, and test your mappings as you go. This makes it easier to troubleshoot errors that pop up.
You will need at least one unique mapping. This unique mapping is how feeds keeps track of nodes. If content on the export site changes, feeds can update existing nodes on the import site with the unique mapping. If it's missing, feeds doesn't know what to match the content to, and it (attempts to) create new content. This is rarely the behavior you want, so make sure you have a unique mapping!
This needs to be something that is, in fact, unique. Ideally, it is also something that doesn't change. A pretty typical technique is to use the NID from the export site, and to import it into a plain text 'GUID' field in the import site. You can hide or delete this field on your import site after you're done. If you have something else that you know is unique, you can also just use that instead. For example, titles may be unique for some content types. For users, the username is unique.
Create and test the feed
To test your feed, you first need to... create it! All we've done so far is create the feed type, but not the feed itself.
So go to Content > Feeds > Add Feed. Name this feed (I usually keep the name consistent with the feed type, as I've never reused a feed type). For the URL, this is the path to your JSON or XML document that you created in the view on your export site.
Click Save and Import.
If all goes according to plan, your feed should think for a moment, and create some new nodes. If you look at those nodes, they should have the data from the export site!
To be real, things almost never go according to plan at this point.
Thankfully, the errors that get thrown at this point are generally pretty good hints at where things went wrong. Sometimes it's something with the view, and sometimes it's an issue with mappings. You may also need to check the reports under Reports > Recent Log Messages for clues.
The best advice I can offer is keep tweaking things on your view and in your mappings, double check your context, and keep importing until things start working. There are a lot of details involved in the process, so a second set of eyes never hurts either.
There are only a few steps left at this point:
- Continue adding mappings until you have all of your fields
- Once everything seems to be working correctly, go back to your original view and change it from only displaying 10 items to displaying all items. Remember, no pagers!
When everything is configured, you can bring over hundreds or thousands of nodes in mere moments. Pat yourself on the back, you've done a great job.
This process works for importing users. There are a few special things you'll need to do/pay attention to:
- You need to create the users as CAS users on the import site before you run your feed importer.
- You can easily do this by creating a view on your export site of just usernames—not an xml feed, you don't even need to save it, you just need a list of all the CAS usernames. Just copy and paste this list into the Add CAS users box.
- When you create your mappings, you'll want to map 'name' from the export site to 'name' on the import site.
- This mapping MUST be unique.
- This is the Drupal username, it matches the CAS username because of how we configure things, but you cannot import into the cas field directly with feeds. This field is how we tell feeds that this user already exists on the import site, and how feeds then knows to import the data to that user rather than create a new user.
Feeds tamper is a useful tool for altering data at the point where it gets imported, but before the node gets created. Some really common uses for it:
- Strip out white space before/after text
- Decode HTML, for example when things like, "See & be seen" should be changed to "See & be seen"
- Find and replace, say when you're changes a field that reads "Dr." to instead read "PhD"
- Import plain text time/date formats
And all sorts of other cool things.
One Last Helpful Hint
Even if you have force update checked, for some reason feeds is almost always sticky on the first re-import. If you import nodes and it says "Nothing has changed," when you know for a fact that things have changed, try just clicking import again. Usually it'll kick into gear on that second import.