Setting up Kaltura - part VI

We'll focus on the Kaltura Drop Folder feature today.  The Drop Folder offers a mechanism whereby an enterprise can bulk upload content without human intervention.  In principle this is an excellent way for a library or archive to ingest many objects into Kaltura without some poor archivist performing individual (or group) uploads via a web GUI.  In practice the mechanism works smoothly when things are going well, but it can be a little difficult to diagnose problems when things go awry.

For example, here's a sample display from the Drop Folders panel from our Kaltura Management Console (KMC), which serves as an all-in-one dashboard for managing content:

Click to see a full-size image

According to this display we have just ingested three items:  an XML file and two video files.  In this particular case the XML file contained all of the metadata for the two video files, and contained instructions that told Kaltura that these were new items to ingest.  The Status field shows a value of Done, and the Error Description field is empty.  This seems good.

We can also see status information if we navigate to the Upload Control panel and select the Bulk Upload view.  Here we see similar info:

Click to see a full-size image

Again, this seems like good news.  The Notification column shows a value of Finished successfully.  Hooray!

But not so fast, my friend....

If we examine one of the video files under the Content panel (Entries tab), we see that none of the extended metadata is present.  We can see the Custom Data fields, but they are all empty.  Hmm, what happened?

If we navigate back to the Upload Control display, the last column offers some possible help:

There is an Action available to download a log file.  That sounds promising.  Let's do that.

The log file is in XML format, and if we open it up in a good browser or text editor or XML editor, we find XML that looks very much like the ingest XML we used in the Drop Folder.  And if we scroll all the way down to the bottom, we find this snippet:

<item><result><errorDescription>customDataItems failed: invalid metadata data: Element 'METXVideoSubmissionElectronicBoardUsed': [facet 'enumeration'] The value ' ' is not an element of the set {'Y', 'N'}. at line 87 Element 'METXVideoSubmissionElectronicBoardUsed': ' ' is not a valid value of the local atomic type. at line 87 </errorDescription></result>

This is telling us that we messed up one of the metadata fields.  If we look at the original ingest XML and find the statement that is supposed to be setting METXVideoSubmissionElectronicBoardUsed, sure enough, there is no value.  (The error occurs on line 296, not 87, which is a bit confusing.)

So the good news is that if we notice the error, we can find a log that will point us at the error.  But detecting the error is a little tricky, and it is easy to see how this would be difficult if we were ingesting, say, 100 items at a time.  So this is not awful, but is also not quite as nice as we might like.

Suggestions:

1. If the XML contains Custom Data, and the Custom Data has errors, but the video still ingests, perhaps a Status of something like "Done with errors" (in the Drop Folders display) or "Finished with Custom Data errors" (in the Bulk Upload Log display).
2. Make the diagnostic message (errorDescription) available without needing to download a file.  This could appear in a new column, or perhaps in a text pop-up.
3. If N - 1 elements of the Custom Data are good, but one is bad, it would be nice if the other N - 1 Custom Data fields are set.  That would make it possible to correct the error manually in the KMC rather than copying fresh XML into the Drop Folder.
4. Suppress the line numbers since they are relative to the log file XML, not the original XML.

Again, overall the Drop Folder feature is very nice, and we will indeed use it to ingest the 20-30,000 video files in our collection.  But since it is likely that we will sometimes make a mistake within the XML (say, forgetting to escape a certain character), it would be great if the KMC would make it hard to detect and diagnose mistakes.