tag:blogger.com,1999:blog-47724317215581533112024-02-21T01:01:41.944-05:00Technology at ICPSRNews and commentary about new technology-related projects under development at ICPSRBryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.comBlogger376125tag:blogger.com,1999:blog-4772431721558153311.post-10370110405385671492013-07-19T06:00:00.000-04:002013-07-19T06:00:04.091-04:00Clients v. customers | services v. products<a href="http://sethgodin.typepad.com/" target="_blank">Seth Godin</a> has another <a href="http://sethgodin.typepad.com/seths_blog/2013/07/clients-vs-customers.html" target="_blank">excellent post</a>. This one notes the distinction between customers (who decide whether or not to buy your product) and clients (who pay you to make things for them).<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://sethgodin.typepad.com/icn.seths.head.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://sethgodin.typepad.com/icn.seths.head.png" height="320" width="167" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Seth Godin</i></td></tr>
</tbody></table>
In the context of ICPSR I think we have a product we call "ICPSR membership." Customers buy it (or not), and if they do, they receive a reasonably well defined set of services, largely centered around the ability to access high quality datasets and documentation. We have many hundreds of customers for this product. I think our Summer Program is also a product, and that too has may hundreds of customers.<br />
<br />
We also have a smaller number, perhaps a dozen or so, of clients. In the best case we have a handful of clients who all pay us to perform a similar set of tasks for them: curate their datasets and documentation, preserve the curated artifacts, and publish the content on a specially "skinned" version of the ICPSR web site for all the world to see. Adding more clients who want us to do this kind of work benefits all of the other clients, and, often, our customers too.<br />
<br />
And like any organization which draws much of its revenue from contract work for clients, we also have those that push us in new, different directions, sometimes for the better, and sometimes for the worse. The trick, of course, is not too try to head off in too many different directions at once. And to favor those clients who pull us in better, not worse, directions.<br />
<br />
<br />Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-15391973369113525082013-07-17T06:00:00.000-04:002013-07-17T06:00:03.735-04:00ICPSR Web Availability - 2012-2013Here are the final numbers for ICPSR's web site availability over our last fiscal year:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCkPvGLh7JNvBICNT_uBR7mJSTiQLNn9U8AApicVd_2GPEswivRvXqzA7_Q6zmQMxqvf3SaO5ERS40ezbKSIxaLfJaG2oVmsqOAFjcArG76tSTjrUd8ujKVLIvFBRUmKM5vkLGa_0FUEKV/s1600/fy_2013_system_availability.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCkPvGLh7JNvBICNT_uBR7mJSTiQLNn9U8AApicVd_2GPEswivRvXqzA7_Q6zmQMxqvf3SaO5ERS40ezbKSIxaLfJaG2oVmsqOAFjcArG76tSTjrUd8ujKVLIvFBRUmKM5vkLGa_0FUEKV/s1600/fy_2013_system_availability.png" height="120" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Click to embiggen</i></td></tr>
</tbody></table>
The year did not start off so well, and we reached the nadir quickly. August 2012 was our worst period of availability in a very bad year for us overall. January, March, and June 2012 also had very poor numbers.<br />
<br />
The main antagonist we faced was a new and unusual problem with our Oracle database server. For many years we would export the content for backup purposes each evening, and it worked well for a decade. However, suddenly in 2012 we began to experience an outage just AFTER each export. Despite intensive analysis by ourselves and local Oracle exports, we never could isolate the root cause of failure.<br />
<br />
We eventually "solved" the problem by exporting our database only once per week v. once per day. That left us more exposed to loss, of course, but it seemed to limit the outages to once per week v. once per day.<br />
<br />
We then replaced the hardware with a new machine with a bit more processor and memory, but with blindingly fast solid-state drives. With the new machine deployed we returned to our daily export schedule, and the machine -- and our web availability -- have been in pretty good shape ever since. The machine went into service in April 2012, and the chart above makes it clear that life has been a little less hectic for our on-call engineer since then.Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-79647589529112701272013-06-21T06:00:00.000-04:002013-06-21T15:00:33.409-04:00A tiny wishlist for Amazon Web Services' Route 53<div class="separator" style="clear: both; text-align: center;">
<a href="https://upload.wikimedia.org/wikipedia/commons/thumb/a/a5/US_53.svg/200px-US_53.svg.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://upload.wikimedia.org/wikipedia/commons/thumb/a/a5/US_53.svg/200px-US_53.svg.png" /></a></div>
We've been using the DNS hosting service, Route 53, from Amazon Web Services (AWS). The default port for a DNS server is UDP (and TCP) 53, and I've always presumed that this was the answer to the question: <i>Why did Amazon name its DNS service Route 53?</i><br />
<br />
In general I like the Route 53 service pretty well. It's smart how the DNS servers listed for a Hosted Zone (the term AWS uses for a domain hosted in Route 53) reside in different top-level domains, like ORG, NET, COM, and even CO.UK. The UI in the AWS Management Console is fine for managing small zones that contain just a handful of records.<br />
<br />
There's one feature that I wish Route 53 had, though, and it would be particularly useful, I think, to research organizations in higher education.<br />
<br />
In our grants and contracts there is often a commitment to build, deploy, and operate some technology deliverable. Often the technology is a web portal of some sort, and the investigator is keen to register a new domain. This leads to an initial registration of something like:<br />
<br />
WhizBangProject.org<br />
<br />
The domain may have only the smallest number of records: an SOA and NS records, of course, and then perhaps an MX record routing mail to a central server, and an A record pointing to the IPv4 address of the web portal.<br />
<br />
Soon, though, the researcher may decide to register the same name in different top-level domains, and we have:<br />
<br />
WhizBangProject.net<br />
<div>
WhizBangProject.com</div>
<div>
WhizBangProject.info<br />
<br />
joining the mix. These domains have EXACTLY the same records as the first one, and so if one is running his/her own DNS service, one can configure the DNS server to use the same zone file when loading all of the domains. This is nice - one file with one set of records to manage for many different domains.<br />
<br />
However, it is often the case that the investigator discovers that the original name is not satisfactory, and so we then register an alternate name in several domains:<br />
<br />
CoolBeansResearch.org</div>
<div>
<div>
CoolBeansResearch.net</div>
</div>
<div>
<div>
CoolBeansResearch.com</div>
</div>
<div>
<div>
CoolBeansResearch.info<br />
<br />
and maybe a slight variant too:<br />
<br /></div>
</div>
<div>
<div>
Cool-Beans-Research.org</div>
<div>
Cool-Beans-Research.net</div>
<div>
Cool-Beans-Research.com</div>
<div>
Cool-Beans-Research.info<br />
<br />
In a world where one runs one's own DNS server, the additional domains are not much extra work. Like the original solution where we pointed the new domains at the same zone file, we can just point these new domains at that same zone file.<br />
<br />
I wish Route 53 would let me create a collection of what they call a Record Set, and then apply those same records to an arbitrary set of what they call Hosted Zones. If the SOA and NS Record Sets were unique to each Hosted Zone, that would be OK; it is really the other records - the ones we add ourselves in Route 53 - that we would want to share across all of the Hosted Zones.</div>
</div>
Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-88704759835302146312013-06-19T06:30:00.000-04:002013-06-19T06:30:01.124-04:00EMC transfer_support_materials fix for anonymous ftp<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJgWJYIjlsC03HOw-hywbYtx9obHuNGeEZlZqtiCOKUGV-AkhvWQ0hKQbK3inm7nOdBvFT-ao6bqvRyDEzQ6dJxFW5m_ewAQqMttydMXwnY4C3KLJBXHD-pjdVCoTulEDOEwgMmXI7NuhT/s1600/img9.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJgWJYIjlsC03HOw-hywbYtx9obHuNGeEZlZqtiCOKUGV-AkhvWQ0hKQbK3inm7nOdBvFT-ao6bqvRyDEzQ6dJxFW5m_ewAQqMttydMXwnY4C3KLJBXHD-pjdVCoTulEDOEwgMmXI7NuhT/s1600/img9.jpg" /></a></div>
Last month I posted about an issue we have been having with our EMC NS 120 NAS. To re-cap briefly... When the NS 120 discovers a problem, one action it often will take is to collect up a bunch of diagnostic information, Zip it up, and then use anonymous ftp to transfer it to EMC. A shell script under the /nas/tools directory called <i>transfer_support_materials</i> does the dirty work. The problem we have been experiencing is rooted in this script; it would fault when trying to transfer the Zip file. <br />
<br />
The sequence of <i>ftp</i> commands inside the script is simple:<br />
<br />
<ol>
<li>Connect to ftp.emc.com</li>
<li>Log in using the user name anonymous and a password unique to the NS 120</li>
<li>change directory to /incoming/APMxxxxxxxxxxx (where the string of x's is replaced with the NS 120 serial number)</li>
<li>transfer the Zip file</li>
</ol>
<div>
The script would always fail at step #3 with the message: <i>File unavailable</i>.</div>
<div>
<br /></div>
<div>
The root of the problem is that the <i>transfer_support_materials</i> script expects the directory to exist, but it doesn't.</div>
<div>
<br /></div>
<div>
At first I thought that the problem was with the EMC anonymous <i>ftp</i> server. I opened several SRs trying to get someone to create the directory. None of the SRs ever reached a satisfactory closure, and I was left with the impression, <i>Of course the directory doesn't exist; we delete them after a couple of days automatically</i>.</div>
<div>
<br /></div>
<div>
So..... The tool to transfer diagnostics expects the directory to exist, and the business process at EMC deletes the directory as a routine matter.</div>
<div>
<br /></div>
<div>
At the suggestion of one of my colleagues, I ran ftp by hand, and discovered that it would happily let me create the directory. That is, I could manually do this:</div>
<div>
<ol>
<li>Connect to ftp.emc.com</li>
<li>Log in using the same credentials as the NS 120</li>
<li>mkdir /incoming/APMxxxxxxxxxx</li>
<li>cd /incoming/APMxxxxxxxxxx</li>
<li>transfer the Zip file</li>
</ol>
</div>
<div>
I decided to tweak <i>transfer_support_materials</i>, adding this new element to the existing sequence of ftp commands. The change is really simple. This:</div>
<div>
<br /></div>
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">#do transfer<br />LFTPCOMMANDFILE="open -u ${username},${password} $HostName;cd $remote_name;rm -f ${newfile##*/};put $newfile"</span><br /><br />becomes this:<br /><br /><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">#do transfer<br />LFTPCOMMANDFILE="open -u ${username},${password} $HostName;<b>mkdir $remote_name;</b>cd $remote_name;rm -f ${newfile##*/};put $newfile"</span><br /><br />Ran a quick test of the script after this change, and <i>Voila!</i>, it works again.<br /> <br />Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-50877760459881518572013-05-01T11:22:00.000-04:002013-05-01T11:22:57.107-04:00EMC anonymous ftp service and transfer_support_materials<div class="separator" style="clear: both; text-align: center;">
<a href="https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcT-crURRr5sOZrXEmjKeH1oC0MhmKFWi8S3CGLpWtJ-NdQdqRn0" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcT-crURRr5sOZrXEmjKeH1oC0MhmKFWi8S3CGLpWtJ-NdQdqRn0" /></a></div>
I have not seen notes about this in forums and boards, and so thought I would pass this along to others who may be using EMC gear.<br />
<br />
About a month ago we had a small problem with one of our NS 120 Celerra NAS units. (It may have been soft errors on one of its disk drives.) The Celerra detected the problem, and went to do its usual thing: collect logs and other analytics, and then copy them to EMC's anonymous ftp site. Our Celerra uses a utility under /nas/tools called /nas/tools/transfer_support_materials to do this. We noticed that when the Celerra tried to transfer the support materials that too failed. And this generated an additional series of critical errors.<br />
<br />
We logged into the Celerra's control station and ran transfer_support_materials by hand. And we saw a message like this:<br />
<br />
<span style="font-family: Courier New, Courier, monospace;">[nasadmin@controller tools]$ /nas/tools/transfer_support_materials -uploadlog<br />transfer_support_materials[12057]: The transfer script has started.<br />PING ftp.emc.com (168.159.219.138) 56(84) bytes of data.<br />From 12.249.233.6 icmp_seq=0 Packet filtered<br /><br />--- ftp.emc.com ping statistics ---<br />1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms<br />, pipe 2<br />cd: Access failed: 550 Requested action not taken. File unavailable. (/incoming/APM00000000000)<br />`/nas/var/emcsupport/support_materials_APM00000000000.130407_1351.zip' at 65536 (0%) 49.1K/s eta:5m [Connection idle]</span><div>
<br /></div>
<div>
I've replaced our Celerra's serial number with the string "00000000000".</div>
<div>
<br /></div>
<div>
We then ran ftp by hand to see if we could replicate the error:</div>
<div>
<br /></div>
<span style="font-family: Courier New, Courier, monospace;">nasadmin@controller tools]$ ftp ftp.emc.com<br />Connected to ftp.emc.com.<br />220-Proceeding further constitutes acknowledgement<br /> to EMC Acceptable Use and Customer Security policies.<br /> Anonymous uploads are immediately moved to a secure server accessible only<br /> within EMC networks.<br /> File downloads from ftp.emc.com are restricted to selected /pub directories, via<br /> temporary secure accounts or via specific permanent secure accounts only.<br /> Anonymous users please login with anonymous and email address as your password<br /> See Powerlink emc278739 for upload instructions.<br /> EMC staff: please refer to current services, FAQ and Best Practices documents at<br /> http://one.emc.com/clearspace/community/active/css/projects/ftp-service<br /> Please email all questions and concerns to ftpquestions@emc.com<br />220 Please reference the FTP Acceptable Use policy: http://itcentral.corp.emc.com/Policies/AcceptableUse.pdf<br />534 Command denied.<br />534 Command denied.<br />KERBEROS_V4 rejected as an authentication type<br />Name (ftp.emc.com:nasadmin): anonymous<br />331 User name okay, need password.<br />Password:<br />230 User logged in, proceed.<br />Remote system type is UNIX.<br />Using binary mode to transfer files.<br />ftp> cd /incoming/APM00000000000<br />550 Requested action not taken. File unavailable.<br />ftp></span><div>
<br /></div>
<div>
So, the problem was that the directory that holds our support materials (/incoming/APM<serialnum>) was missing or had its mode set to something that disallowed access.</div>
<div>
<br /></div>
<div>
We contacted EMC, and some days later they confirmed that the problem was indeed that the directory was missing, and that they had recreated it. We then ran ftp by hand to confirm that everything was working again, and it was. That was good news, but when we tried the same thing on our second NS 120 Celerra, we discovered that it too was missing its "support directory" on the ftp server. So we added that trouble report to our service request, and some days later, EMC confirmed that too had been missing, and then again recreated it. In speaking with EMC it is a bit unclear if this problem is particular to us or more broad.</div>
<div>
<br /></div>
<div>
The upshot of the story is that if you too run a Celerra or other product that sends support materials to EMC via anonymous ftp, this might be a good day to test out transfer_support_materials to make sure that your "support directory" is intact. If so, that's great, but if it is missing, you may want to open a service request with EMC soon so that they can recreate the directory for you. Better to have it in place before your system needs to send support materials, but is not able to do so.</div>
<div>
<br /></div>
<div>
I should note that we're still happy overall with EMC; in fact, we've just purchased the first three nodes of a new Isilon storage system from them. So the intent here isn't to excoriate them over the missing ftp directory; it was easy to reproduce the problem and to correct it. But we did wish that we had been able to learn about the problem prior to the disk failure so that it could have been corrected earlier, not when the Celerra was trying to report a disk failure.</div>
<div>
<div>
<br /></div>
<div>
<br /></div>
</div>
Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-63504989146750540502013-04-29T05:00:00.000-04:002013-04-29T05:00:07.174-04:00ICPSR launches Measures of Effective Teaching web siteSome of my colleagues, including ICPSR Director George Alter, gave a demo of one of our newest Web sites and collections at the <a href="http://www.aera.net/Home/tabid/10041/Default.aspx" target="_blank">American Educational Research Association</a> 2013 <a href="http://www.aera.net/tabid/10208/Default.aspx" target="_blank">annual meeting</a> on Sunday.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://www.icpsr.umich.edu/icpsrweb/METLDB/index.jsp" style="margin-left: auto; margin-right: auto;" target="_blank"><img alt="MET LDB web site" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhQ1QnfgMldjr_0H1QNGytdlKDJ9VAjrB6KW97jqlEpzS2OfZ4EdfGOUP2lei6Za3djNui5gqn1QoT2xuExKcdSO6sPC4IRL1sNJB8Fege8PIcNKY6S2zFDjHOG4dEZrH1oG3zJLVWa7hcw/s1600/Screen+shot+2013-04-28+at+3.39.48+PM.png" height="255" title="MET LDB web site" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"><i>Click the image to navigate to the live site</i></td></tr>
</tbody></table>
My team has built the video portal portion of the system. The portal enables a researcher to play a list of videos that s/he has selected to view based on an analysis of the associated quantitative data and tagging data. Access to the video and datasets is restricted and requires one to complete a data use agreement via ICPSR's web-based request system.<br />
<br />
We're grateful for the support we've received from the <a href="http://www.gatesfoundation.org/" target="_blank">Bill and Melinda Gates Foundation</a> to make all of this possible.Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com2tag:blogger.com,1999:blog-4772431721558153311.post-68019795005252672792013-04-22T07:00:00.000-04:002013-04-22T07:00:01.369-04:00Qualys browser checker<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiShxgxQhyvvMEb0bEEWnbcPL5h7a4ZTWZmcfmrZUm4UT6CTELNbHkCfMXSFgmTW1knETHsIDWuQpBTJoThHzwpiygKP0ut5p9EJxSZHgSt4nmggUYfRTNPxuHo7_y6fGVQZTs6woj9oDXM/s1600/Screen+Shot+2013-04-20+at+8.00.44+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiShxgxQhyvvMEb0bEEWnbcPL5h7a4ZTWZmcfmrZUm4UT6CTELNbHkCfMXSFgmTW1knETHsIDWuQpBTJoThHzwpiygKP0ut5p9EJxSZHgSt4nmggUYfRTNPxuHo7_y6fGVQZTs6woj9oDXM/s1600/Screen+Shot+2013-04-20+at+8.00.44+PM.png" height="119" width="320" /></a></div>
Ever since the recent craziness with vulnerabilities in Java plugins, I've making a concious effort to use Qualys's browser checker - <a href="https://browsercheck.qualys.com/">https://browsercheck.qualys.com/</a> - on a routine basis both at home and at the office. <br />
<br />
Installing the tool in your browser is very easy, and the service is free and painless to use. I have been using it to both to determine if my current browser and plugins are up to date, and also to identify plugins that are installed and enabled, but which I don't really need or use (e.g., Silverlight which I often disable for long stretches at a time).<br />
<br />
Qualys generates a nice report<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBZvV3-Z0e6uWHmVyNsQF4ciYvRHkMertt45uD-O7jp-JlvFSRiIrDr9_GVu6rkHsrXChyphenhyphenL2drOG86clWSDJClQOiJa1rkWIqre4mjpb64nwfiwGbLHxfdMBbs-ROAygrZ3_Abb0T_p-eY/s1600/Screen+Shot+2013-04-20+at+8.03.20+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBZvV3-Z0e6uWHmVyNsQF4ciYvRHkMertt45uD-O7jp-JlvFSRiIrDr9_GVu6rkHsrXChyphenhyphenL2drOG86clWSDJClQOiJa1rkWIqre4mjpb64nwfiwGbLHxfdMBbs-ROAygrZ3_Abb0T_p-eY/s1600/Screen+Shot+2013-04-20+at+8.03.20+PM.png" height="320" width="316" /></a></div>
<br />
<br />
like the one above to let you know if everything is up-to-date.Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-67561597124735274342013-04-18T09:16:00.001-04:002013-04-18T09:16:25.836-04:00Web availability at ICPSR - March 2013ICPSR's content delivery system showed very high availability in March 2013: a bit over 99.95% uptime. We had only two problems in March. One was a power outage that affected our headquarters on the University of Michigan campus, and we experienced a small amount of downtime as we moved service to our replica in Amazon's cloud. The second was a 21-minute outage due to a continuing -- but now solved, we think -- problem with exporting content from our Oracle database server.<br />
<br />
Here are the overall numbers for ICPSR's 2012-2013 fiscal year:<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEht4QrakifvODnMyyxJJdssdPx_UvsAKYaVUtHJZpT1SNXKIDMuemZCy7wPdVTwPmvuQxDmLtZemfmhs037q80uoWaqw2qcRaAs_4AMGmxOV-cOyvKUqqEulyXAesJl6CYAH1ooMVhaoaN8/s1600/fy_2013_system_availability.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="120" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEht4QrakifvODnMyyxJJdssdPx_UvsAKYaVUtHJZpT1SNXKIDMuemZCy7wPdVTwPmvuQxDmLtZemfmhs037q80uoWaqw2qcRaAs_4AMGmxOV-cOyvKUqqEulyXAesJl6CYAH1ooMVhaoaN8/s400/fy_2013_system_availability.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">click to enlarge</td></tr>
</tbody></table>
<br />
We replaced our aging Oracle database server with a new machine which has twice the memory, twice the computing power, and perhaps most impressively, has 300 times the disk I/O speed(!). The new machine has an array of solid-state drives (SSDs), and we use this for all of our database storage. (The operating system resides on conventional disk drive technology.)Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-31263506524455206452012-11-16T06:00:00.000-05:002012-11-16T06:00:06.133-05:00Web availability at ICPSR - October 2012October was a very good month for system uptime - over 99.9% availability:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivBWIdA4dSZGhBOL3UYmjP-dyCpWa3aCUqtjpuAKM-sWQy7yy00EPDwsKbLll_tcoaTINprJ8zlVftu1RXZQv8y3RSMmfH0lKubngMTGAyKUsVcSP36ajVQrKIVUlWUwtEubUc1hSFuHeg/s1600/fy_2013_system_availability(1).png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEivBWIdA4dSZGhBOL3UYmjP-dyCpWa3aCUqtjpuAKM-sWQy7yy00EPDwsKbLll_tcoaTINprJ8zlVftu1RXZQv8y3RSMmfH0lKubngMTGAyKUsVcSP36ajVQrKIVUlWUwtEubUc1hSFuHeg/s1600/fy_2013_system_availability(1).png" height="120" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Click chart to enlarge</td></tr>
</tbody></table>
That's good news after a much rougher September. So far things look good this month, although a number of very short-lived outages have already pushed us below 99.9% for the month.Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-84556411547527064222012-11-14T07:00:00.000-05:002012-11-14T07:00:12.632-05:00A commentary on MOOCs from Clay ShirkySome of my colleagues - past and present - are attending classes in Massive Open Online Courses (MOOCs). I've been following their stories and also columnists who have been talking about MOOCs and education. It is a very interesting time.<br />
<br />
Clay Shirky has a long post (<a href="http://www.shirky.com/weblog/2012/11/napster-udacity-and-the-academy/" target="_blank">Napster, Udacity, and the Academy</a>) about MOOCs that is well worth reading. Some highlights:<br />
<blockquote class="tr_bq">
<i>The recording industry concluded this new audio format would be no threat, because quality mattered most. Who would listen to an MP3 when they could buy a better-sounding CD at the record store? Then Napster launched, and quickly became the fastest-growing piece of software in history. The industry sued Napster and won, and it collapsed even more suddenly than it had arisen.<br /><br />If Napster had only been about free access, control of legal distribution of music would then have returned the record labels. That’s not what happened. Instead, Pandora happened. Last.fm happened. Spotify happened. ITunes happened. Amazon began selling songs in the hated MP3 format.</i></blockquote>
and<br /><blockquote class="tr_bq">
<i>It’s been interesting watching this unfold in music, books, newspapers, TV, but nothing has ever been as interesting to me as watching it happen in my own backyard. Higher education is now being disrupted; our MP3 is the massive open online course (or MOOC), and our Napster is Udacity, the education startup.<br /><br />We have several advantages over the recording industry, of course. We are decentralized and mostly non-profit. We employ lots of smart people. We have previous examples to learn from, and our core competence is learning from the past. And armed with these advantages, we’re probably going to screw this up as badly as the music people did.</i></blockquote>
Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-47426762248860319702012-11-12T16:02:00.003-05:002012-11-12T16:02:53.803-05:00Nick Carr, MOOCs, and ethicsIn his post <i><a href="http://www.roughtype.com/?p=2005" target="_blank">The ethics of MOOC research</a></i>, Nick Carr describes a note he received from a colleague in academia who comments on the research agenda of Massive Open Online Courses:<br />
<blockquote class="tr_bq">
<i><span style="background-color: white; color: #333333; font-family: Georgia, 'Bitstream Charter', serif; font-size: 14px; line-height: 23px;">The MOOCs’ research agenda seems entirely wholesome. But it does raise some tricky ethical issues, as a correspondent from academia pointed out to me after my article appeared. “At most institutions,” he wrote, the kind of behavioral research the MOOCs are doing “would qualify as research on human subjects, and it would have to be approved and monitored by an institutional review board, yet I have heard nothing about that being the case with this new adventure in technology.” Universities are, for good reason, </span><a href="http://hrpp.umich.edu/" style="background-color: white; border: 0px; color: #333333; font-family: Georgia, 'Bitstream Charter', serif; font-size: 14px; line-height: 23px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">very careful</a><span style="background-color: white; color: #333333; font-family: Georgia, 'Bitstream Charter', serif; font-size: 14px; line-height: 23px;"> about regulating, approving, and monitoring biological and behavioral research involving human subjects. In addition to the general ethical issues raised by such studies, there are strict </span><a href="http://www.hhs.gov/ohrp/" style="background-color: white; border: 0px; color: #333333; font-family: Georgia, 'Bitstream Charter', serif; font-size: 14px; line-height: 23px; margin: 0px; outline: 0px; padding: 0px; vertical-align: baseline;">federal regulations</a><span style="background-color: white; color: #333333; font-family: Georgia, 'Bitstream Charter', serif; font-size: 14px; line-height: 23px;"> governing them. I am no expert on this subject, but my quick reading of some of the federal regulations suggests that certain kinds of purely pedagogical research are exempt from the government rules, and it may well be that the bulk of the MOOC research falls into that category.</span></i></blockquote>
Given the intense energy ICPSR has been putting its systems for protecting confidential research data and facilitating requests for using such data, I found this very interesting. <br />
<br />
I see parallels here with collecting and using personal information. If one conducts a survey and asks personal questions to well-consented adults, the results might one day become an interesting, restricted-use dataset. But if the same information is harvested from freely and openly blogs, tweets, and wall posts, would it also become restricted-use data?Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com1tag:blogger.com,1999:blog-4772431721558153311.post-31948128186739921002012-10-15T06:30:00.000-04:002012-10-15T06:30:03.250-04:00Artificial Intelligence as defined by Nick CarrNick Carr has a short post <a href="http://www.roughtype.com/?p=1915" target="_blank">here</a> marking the occasion of Facebook's one billionth member. He goes on to talk a bit about some work at Google on neural nets, but then includes this gem on artificial intelligence:<br />
<blockquote class="tr_bq">
<i>Forget the Turing Test. We’ll know that computers are really smart when computers start getting bored. If you assign a computer a profoundly tedious task like spotting potential house numbers in video images, and then you come back a couple of hours later and find that the computer is checking its Facebook feed or surfing porn, then you’ll know that artificial intelligence has truly arrived.</i></blockquote>
It's a short post and a good read.Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com1tag:blogger.com,1999:blog-4772431721558153311.post-39503081674172003032012-10-10T07:00:00.000-04:002012-10-10T07:00:19.903-04:00September 2012 deposits at ICPSRThe numbers from September are in:<br />
<br />
<br />
<table align="center" border="2">
<tbody>
<tr><td># of files</td><td># of deposits</td><td>File format</td></tr>
<tr><td>29</td><td>1</td><td>F 0x07 video/h264</td></tr>
<tr><td>145</td><td>17</td><td>application/msword</td></tr>
<tr><td>5</td><td>1</td><td>application/octet-stream</td></tr>
<tr><td>292</td><td>11</td><td>application/pdf</td></tr>
<tr><td>14</td><td>5</td><td>application/vnd.ms-excel</td></tr>
<tr><td>1</td><td>1</td><td>application/vnd.ms-powerpoint</td></tr>
<tr><td>120</td><td>1</td><td>application/x-arcview</td></tr>
<tr><td>31</td><td>1</td><td>application/x-dbase</td></tr>
<tr><td>1</td><td>1</td><td>application/x-rar</td></tr>
<tr><td>24</td><td>6</td><td>application/x-sas</td></tr>
<tr><td>14</td><td>6</td><td>application/x-spss</td></tr>
<tr><td>13</td><td>5</td><td>application/x-stata</td></tr>
<tr><td>5</td><td>5</td><td>application/x-zip</td></tr>
<tr><td>1</td><td>1</td><td>image/jpeg</td></tr>
<tr><td>32</td><td>1</td><td>image/x-3ds</td></tr>
<tr><td>70</td><td>2</td><td>multipart/appledouble</td></tr>
<tr><td>10</td><td>3</td><td>text/plain; charset=unknown</td></tr>
<tr><td>62</td><td>18</td><td>text/plain; charset=us-ascii</td></tr>
<tr><td>2</td><td>1</td><td>text/rtf</td></tr>
<tr><td>29</td><td>2</td><td>text/xml</td></tr>
</tbody></table>
<br />
Interesting month in that we have the usual stuff in the usual quantities, but we also have a large number of unusual formats hitting the doorstep, such as ArcView and Apple Double. And we also have a usual format in an unusually high quantity (MS Word).Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-9708304902724910142012-10-05T06:30:00.000-04:002012-10-05T06:30:01.365-04:00September 2012 web availabilitySeptember was an OK, but not great month for web availability:<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><img height="120" src="https://docs.google.com/a/umich.edu/spreadsheet/oimg?key=0AgUA8KteqtijdGx4cXl5UHl4M2Vmdi1Rc1Bya2NzZXc&oid=4&zx=r69ure49q75i" style="margin-left: auto; margin-right: auto;" width="400" /></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Click to enlarge</td></tr>
</tbody></table>
We eliminated one frequent, but short-lived source of downtime when we stopped exporting the content of our Oracle database nightly. We are now doing it only on the weekend, and while that adds some risk, we're gaining significant uptime. (For some reason that we do not understand, our Oracle instance stops answering queries for 15-20 minutes about ten minutes AFTER the export completes.) We have a new server racked and ready to install, and we're hoping that a fast new machine with solid-state drives will solve the problem for us.<br />
<br />
We did run into some trouble mid-month when some routine maintenance went awry, and we had to fail over to our replica over the weekend of September 15 and 16. The total amount of downtime was about 90 minutes total over the course of the weekend, but the replica kept the problem from clobbering our service completely.<br />
<br />
After that we had pretty smooth sailing for the rest of the month. Just 16 minutes of downtime for the rest of the month.Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-47150264821892771942012-10-03T06:00:00.000-04:002012-10-03T06:00:18.112-04:00Setting up Kaltura - part VIWe'll focus on the Kaltura Drop Folder feature today. The Drop Folder offers a mechanism whereby an enterprise can bulk upload content without human intervention. In principle this is an excellent way for a library or archive to ingest many objects into Kaltura without some poor archivist performing individual (or group) uploads via a web GUI. In practice the mechanism works smoothly when things are going well, but it can be a little difficult to diagnose problems when things go awry.<br />
<br />
For example, here's a sample display from the Drop Folders panel from our Kaltura Management Console (KMC), which serves as an all-in-one dashboard for managing content:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8HlnslhEEiG4uSWpVWqtGgUtMy_FuReodEwweeANyzlsvMSsQ0qcIVxYMX42ez5KE2OZixm8DK5k1E4XZU7JXI8J4vHM9zc_jgu5SWZTw4WX0J90yScA3eLA8a2DZHisJUSGuj0sXlaG3/s1600/Screen+Shot+2012-09-28+at+8.22.51+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="61" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8HlnslhEEiG4uSWpVWqtGgUtMy_FuReodEwweeANyzlsvMSsQ0qcIVxYMX42ez5KE2OZixm8DK5k1E4XZU7JXI8J4vHM9zc_jgu5SWZTw4WX0J90yScA3eLA8a2DZHisJUSGuj0sXlaG3/s400/Screen+Shot+2012-09-28+at+8.22.51+PM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Click to see a full-size image</td></tr>
</tbody></table>
<br />
According to this display we have just ingested three items: an XML file and two video files. In this particular case the XML file contained all of the metadata for the two video files, and contained instructions that told Kaltura that these were new items to ingest. The Status field shows a value of Done, and the Error Description field is empty. This seems good.<br />
<br />
We can also see status information if we navigate to the Upload Control panel and select the Bulk Upload view. Here we see similar info:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTACIW60XjZPxTHltePdkbnvqp0v2oUJBfCqixaVAXckgVltoU7UMAIdS2ld1gAM-P6-cracxFHJENSGORPpXGumlmSYi2mXfvKIt50SenG9Q8qjf-csKr4KDJ-7g8RDPOpRJxW6AQXK5H/s1600/Screen+Shot+2012-09-28+at+8.25.40+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="27" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhTACIW60XjZPxTHltePdkbnvqp0v2oUJBfCqixaVAXckgVltoU7UMAIdS2ld1gAM-P6-cracxFHJENSGORPpXGumlmSYi2mXfvKIt50SenG9Q8qjf-csKr4KDJ-7g8RDPOpRJxW6AQXK5H/s400/Screen+Shot+2012-09-28+at+8.25.40+PM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Click to see a full-size image</td></tr>
</tbody></table>
Again, this seems like good news. The Notification column shows a value of Finished successfully. Hooray!<br />
<br />
But not so fast, my friend....<br />
<br />
If we examine one of the video files under the Content panel (Entries tab), we see that none of the extended metadata is present. We can see the Custom Data fields, but they are all empty. Hmm, what happened?<br />
<br />
If we navigate back to the Upload Control display, the last column offers some possible help:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIVXhe37tOj1H-_LeiCk-ndpM08dj42mRAJYtSwcSexwYGxzx0ys74C1ogOSYDl6V2LTIKnoP-02JAHS4FJ_cNjVEgwGtnqT7ztZq_19QgkZn_J_xb2701Rs-krLNtfAU5aO9EGoYNTXgN/s1600/Screen+Shot+2012-09-28+at+8.29.10+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIVXhe37tOj1H-_LeiCk-ndpM08dj42mRAJYtSwcSexwYGxzx0ys74C1ogOSYDl6V2LTIKnoP-02JAHS4FJ_cNjVEgwGtnqT7ztZq_19QgkZn_J_xb2701Rs-krLNtfAU5aO9EGoYNTXgN/s1600/Screen+Shot+2012-09-28+at+8.29.10+PM.png" /></a></div>
<br />
There is an Action available to download a log file. That sounds promising. Let's do that.<br />
<br />
The log file is in XML format, and if we open it up in a good browser or text editor or XML editor, we find XML that looks very much like the ingest XML we used in the Drop Folder. And if we scroll all the way down to the bottom, we find this snippet:<br />
<br />
<blockquote class="tr_bq">
<i><item><result><errorDescription>customDataItems failed: invalid metadata data: </i><i>Element 'METXVideoSubmissionElectronicBoardUsed': [facet 'enumeration'] </i><i>The value ' ' is not an element of the set {'Y', 'N'}.</i><i> at line 87 </i><i>Element 'METXVideoSubmissionElectronicBoardUsed': </i><i>' ' is not a valid value of the local atomic type.</i><i> at line 87 </errorDescription></result></i></blockquote>
This is telling us that we messed up one of the metadata fields. If we look at the original ingest XML and find the statement that is supposed to be setting METXVideoSubmissionElectronicBoardUsed, sure enough, there is no value. (The error occurs on line 296, not 87, which is a bit confusing.)<br />
<br />
So the good news is that if we notice the error, we can find a log that will point us at the error. But detecting the error is a little tricky, and it is easy to see how this would be difficult if we were ingesting, say, 100 items at a time. So this is not awful, but is also not quite as nice as we might like. <br />
<br />
Suggestions:<br />
<ol>
<li>If the XML contains Custom Data, and the Custom Data has errors, but the video still ingests, perhaps a Status of something like "Done with errors" (in the Drop Folders display) or "Finished with Custom Data errors" (in the Bulk Upload Log display).</li>
<li>Make the diagnostic message (errorDescription) available without needing to download a file. This could appear in a new column, or perhaps in a text pop-up.</li>
<li>If N - 1 elements of the Custom Data are good, but one is bad, it would be nice if the other N - 1 Custom Data fields are set. That would make it possible to correct the error manually in the KMC rather than copying fresh XML into the Drop Folder.</li>
<li>Suppress the line numbers since they are relative to the log file XML, not the original XML.</li>
</ol>
<div>
Again, overall the Drop Folder feature is very nice, and we will indeed use it to ingest the 20-30,000 video files in our collection. But since it is likely that we will sometimes make a mistake within the XML (say, forgetting to escape a certain character), it would be great if the KMC would make it hard to detect and diagnose mistakes.</div>
Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-1561393117647904522012-10-01T06:30:00.000-04:002012-10-01T06:30:00.207-04:00ICPSR Director of Curation ServicesWe have interviewed all of the candidates.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEcf0TMBIOQVGLSvU8uDAeOPkxjUK8k1qfTPwYo77LNnMyHwe9AOwlnNMzbB1XxX2KZ4ZdR_I95XCctdunKoKDEzcGeUNVRdWp6dlSHuA9HtET4LRy9Pc8ID8soSD7HTilPSeRe1sHyqpm/s1600/Screen+Shot+2012-09-26+at+3.59.02+PM.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEcf0TMBIOQVGLSvU8uDAeOPkxjUK8k1qfTPwYo77LNnMyHwe9AOwlnNMzbB1XxX2KZ4ZdR_I95XCctdunKoKDEzcGeUNVRdWp6dlSHuA9HtET4LRy9Pc8ID8soSD7HTilPSeRe1sHyqpm/s1600/Screen+Shot+2012-09-26+at+3.59.02+PM.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">See <a href="http://dilbert.com/strips/comic/2011-10-30">http://dilbert.com/strips/comic/2011-10-30</a> for the full cartoon</td></tr>
</tbody></table>
I had the opportunity to meet with several of the candidates, and we have several excellent ones. With a little bit of luck we should be able to announce who will be filling the position sometime soon.<br />
<br />
And then s/he can explain exactly what Curation Services are. :-)Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-80176783389021667972012-09-28T07:00:00.000-04:002012-09-28T07:00:13.730-04:00Be careful when answering, "Both"When I am working with someone to work through the requirements of a new system or project, I will often ask a series of questions that help shape my understanding of what the person wants. Often these fall into a pattern where I ask a series of either/or questions, like this:<br />
<br />
<i>Do you want it optimized for security <b>or</b> ease of use?</i><br />
<i><br /></i>
<i>Does this fit into a wholesale <b>or</b> retail delivery paradigm?</i><br />
<i><br /></i>
<i>Will this be used by external customers <b>or</b> internal staff?</i><br />
<i><br /></i>
<i>Is this intended to drive new revenues <b>or</b> decrease current costs?</i><br />
<br />
In many ways this is like going to the ophthalmologist who has you look through lens A and then lens B, and then asks the question, "Which was better, A or B?" Both of us are trying to bring the problem into focus.<br />
<br />
The single most dreaded (by me) answer to these questions is: <i>Both</i>.<br />
<br />
In some cases this answer really means, "I am not sure what I want." Or, "I'm too busy to think about this." Or, "I don't care."<br />
<br />
This, obviously, does not help when gathering requirements. And so it is a real barrier to scoping the project. Sometimes, of course, an answer of <i>Both</i> is a fine start to a longer answer.<br />
<blockquote class="tr_bq">
<i>We really do need both in this case. We want to build a system for managing metadata that can be used by both the staff and external people equally easily. We are changing our entire workflow so that either population can manage our metadata, and this is our new business practice.</i></blockquote>
That is a fine use of <i>Both</i>. In fact, if the person gave a different answer, we might needlessly limit the usefulness of the system we build.<br />
<br />
And another fine answer, just like we sometimes tell the ophthalmologist is <i>I don't know</i>. There's nothing wrong with that answer. However, just like with the ophthalmologist, when I hear this answer, I reach into my bag of lens, and try another pair to bring the issue into focus.Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-8831811471853075432012-09-26T06:30:00.000-04:002012-09-26T06:30:04.615-04:00Kaltura pilot at the U-M and ICPSRThe University of Michigan in-house newspaper/newsletter ran a <a href="http://www.ur.umich.edu/update/archives/120830/vcm" target="_blank">nice piece</a> on the Video Contement Management pilot (using Kaltura), which includes our Bill and Melinda Gates Foundation project: Measures of Effective Teaching - Extenstion.<br />
<br />
Here is the part about us:<br />
<blockquote class="tr_bq">
<i>• ISR and the School of Education are engaged in a collaborative
research project in which a large collection of video assets is a
primary data-type. This data will become a shared repository available
to research partners at universities, public agencies, and private
foundations.</i></blockquote>
The timing here was perfect for us since we were in the market for a system to manage and stream about 20TB of video to thousands of simultaneous users.Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-25298889413517250062012-09-24T06:00:00.000-04:002012-09-24T06:00:05.034-04:00DuraSpace announces SDSC as storage partner<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbpKl0k3w9RdMQPiDyyb2wvdA2iUPFJyX4JmLy0puouLndQJ3AUyCirEqJCwK5ltIDtT7NRuNLNjIgWzp3zXg_vUVDexzP-Ec814H03KNC-3QRWgMzYvNv_5af1E8CFcidRcaunuwyyFTz/s1600/images.jpeg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbpKl0k3w9RdMQPiDyyb2wvdA2iUPFJyX4JmLy0puouLndQJ3AUyCirEqJCwK5ltIDtT7NRuNLNjIgWzp3zXg_vUVDexzP-Ec814H03KNC-3QRWgMzYvNv_5af1E8CFcidRcaunuwyyFTz/s1600/images.jpeg" /></a></div>
DuraSpace <a href="http://duraspace.org/duraspace-and-sandiego-supercomputer-center-sdsc-partner-provide-cost-effective-cloud-storage-and-pr" target="_blank">announced</a> recently their relationship with SDSC as a storage provider for DuraCloud. As I <a href="http://techaticpsr.blogspot.com/2012/07/amazons-loss-is-sdscs-gain.html" target="_blank">posted</a> a while ago, we have been using both DuraCloud and their SDSC storage partner for a while. It's great to see DuraCloud continue to grow.Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-62969301807029741712012-09-21T06:30:00.000-04:002012-09-21T06:30:03.512-04:00Setting up Kaltura - part VIn this post we will look at the XML we use to ingest content into Kaltura through its Drop Folder mechanism. To re-cap an <a href="http://techaticpsr.blogspot.com/2012/09/setting-up-kaltura-part-iv.html" target="_blank">earlier post</a> about the Drop Folder, ours is a subdirectory under the home directory of the 'kaltura' user. We provisioned this account on a special-purpose machine that Kaltura accesses via <i>sftp</i> to fetch content without human intervention.<br />
<br />
Kaltura has a <a href="http://www.kaltura.com/api_v3/xsdDoc/?page=overview" target="_blank">pretty nice guide</a> to building the XML, which looks an awful lot like Media RSS. And they also make some short examples available, but we always find it useful to have a real-world example. Here's ours.<br />
<br />
Preface: Our stuff is a little unusual. That is:<br />
<ol>
<li>We always have pairs of videos, one classroom and one blackboard</li>
<li>We have lots of metadata and it applies to both videos, and so the metadata gets repeated in the XML. I have excised much of the metadata in our XML for this post</li>
<li>All of the metadata is fake; it is not real metadata about an actual classroom video</li>
<li>You can find a copy of the XML that we diagram below at this URL <a href="http://goo.gl/OxbyU">http://goo.gl/OxbyU</a></li>
<li>Our use of Kaltura is in support of the Measures of Effective Teaching (Extension) project, and so there are many references to 'metext' in the metadata</li>
<li>We will be generating the XML for ingest programatically</li>
</ol>
<div>
Here goes.</div>
<div>
<br /></div>
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzvgkttWYdsRb0kHs13FjGZQZp_UtVNpMhhG1kfcu3lARamySZ_-beJo3llo4V1JLsEIP2O2vBwoGHhTFTMn9ahGODLnlf-zVVLPpCNYXqqHSX_42YI63VgYy9zdVGJt_7Lyh8FIjW6vhM/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <?xml version="1.0"?>
<mrss xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="ingestion.xsd">
<channel>
<item>
<!-- These do not change -->
<action>add</action>
<type>1</type>
<!-- This changes for each video -->
<referenceId>12345-Board-video-rendition-MP4-H264v-Standard.mp4</referenceId>
<!-- This does not change -->
<userId>metext</userId>
<!-- This changes for each video -->
<name>RCN 12345 board video</name>
<!-- This changes for each video -->
<description>RCN 12345 board video</description>
</code></pre>
<br />
We have the usual XML stuff at the beginning, and then the start of the Media RSS.<br />
<br />
Action = add for new content.<br />
<br />
Type = 1 for video content.<br />
<br />
ReferenceID is the name of the original file.<br />
<br />
UserID is the pseudo-user who will be linked to the content in Kaltura.<br />
<br />
Name and Description are exposed as base metadata in Kaltura.<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzvgkttWYdsRb0kHs13FjGZQZp_UtVNpMhhG1kfcu3lARamySZ_-beJo3llo4V1JLsEIP2O2vBwoGHhTFTMn9ahGODLnlf-zVVLPpCNYXqqHSX_42YI63VgYy9zdVGJt_7Lyh8FIjW6vhM/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <!-- Always assign two tags, one called metext and the other board or classroom -->
<tags>
<tag>metext</tag>
<tag>board</tag>
</tags>
<!-- This does not change -->
<categories>
<category>metext</category>
</categories>
<!-- This does not change -->
<media>
<mediaType>1</mediaType>
</media>
</code></pre>
<br />
We tag everything with the project name and indicate whether it is a video of the blackboard or the classroom.<br />
<br />
Kaltura uses Categories as a main way to browse and find content. We treat this as if it were a type of "is in collection" sort of attribute.<br />
<br />
<br />
MediaType = 1 for video.<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzvgkttWYdsRb0kHs13FjGZQZp_UtVNpMhhG1kfcu3lARamySZ_-beJo3llo4V1JLsEIP2O2vBwoGHhTFTMn9ahGODLnlf-zVVLPpCNYXqqHSX_42YI63VgYy9zdVGJt_7Lyh8FIjW6vhM/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <!-- This changes for each video -->
<contentAssets>
<content>
<dropFolderFileContentResource filePath="12345-Board-video-rendition-MP4-H264v-Standard.mov"/>
</content>
</contentAssets>
</code></pre>
<br />
This tells Kaltura that the video file is in the Drop Folder along with the XML. <br />
<br />
Now for our project-specific metadata, which fits into a Kaltura structure called Custom Data:<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzvgkttWYdsRb0kHs13FjGZQZp_UtVNpMhhG1kfcu3lARamySZ_-beJo3llo4V1JLsEIP2O2vBwoGHhTFTMn9ahGODLnlf-zVVLPpCNYXqqHSX_42YI63VgYy9zdVGJt_7Lyh8FIjW6vhM/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> <!-- This changes for each video -->
<customDataItems>
<customData metadataProfileId="22971">
<xmlData>
<metadata>
<METXDistrictDistrictName>Ann Arbor</METXDistrictDistrictName>
<METXDistrictDistrictNum>20</METXDistrictDistrictNum>
<METXSchoolSchoolName>Huron High School</METXSchoolSchoolName>
<METXSchoolSchoolMETXID>33</METXSchoolSchoolMETXID>
<!-- Kaltura wants the date to be in xs:long format, that is, seconds from the epoch -->
<METXVideoSubmissionCaptureDate>1344440267</METXVideoSubmissionCaptureDate>
</metadata>
</xmlData>
</customData>
</customDataItems>
</code></pre>
<br />
The ID attribute is from our Kaltura KMC. I had to create the Custom Data schema first, and then reference it in the ingest XML here.<br />
<br />
Most of the metadata fields are simple strings or strings from a controlled vocabulary. We do have one date item, and sadly Kaltura expects it to be in a difficult-to-use format, seconds since the epoch.<br />
<br />
After this section of the XML is a closing tag for item, and then the whole thing repeats with only minor variation for the classroom video. I'll include it below for completeness.<br />
<br />
<br />
<pre style="background-image: URL(https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzvgkttWYdsRb0kHs13FjGZQZp_UtVNpMhhG1kfcu3lARamySZ_-beJo3llo4V1JLsEIP2O2vBwoGHhTFTMn9ahGODLnlf-zVVLPpCNYXqqHSX_42YI63VgYy9zdVGJt_7Lyh8FIjW6vhM/s320/codebg.gif); background: #f0f0f0; border: 1px dashed #CCCCCC; color: black; font-family: arial; font-size: 12px; height: auto; line-height: 20px; overflow: auto; padding: 0px; text-align: left; width: 99%;"><code style="color: black; word-wrap: normal;"> </item>
<item>
<!-- These do not change -->
<action>add</action>
<type>1</type>
<!-- This changes for each video -->
<referenceId>12345-Classroom-video-rendition-MP4-H264v-Standard.mp4</referenceId>
<!-- This does not change -->
<userId>metext</userId>
<!-- This changes for each video -->
<name>RCN 12345 classroom video</name>
<!-- This changes for each video -->
<description>RCN 12345 classroom video</description>
<!-- This changes for each video -->
<tags>
<tag>metext</tag>
<tag>classroom</tag>
</tags>
<!-- This does not change -->
<categories>
<category>metext</category>
</categories>
<!-- This does not change -->
<media>
<mediaType>1</mediaType>
</media>
<!-- This changes for each video -->
<contentAssets>
<content>
<dropFolderFileContentResource filePath="12345-Classroom-video-rendition-MP4-H264v-Standard.mov"/>
</content>
</contentAssets>
<!-- This changes for each video -->
<customDataItems>
<customData metadataProfileId="22971">
<xmlData>
<metadata>
<METXDistrictDistrictName>Ann Arbor</METXDistrictDistrictName>
<METXDistrictDistrictNum>20</METXDistrictDistrictNum>
<METXSchoolSchoolName>Huron High School</METXSchoolSchoolName>
<METXSchoolSchoolMETXID>33</METXSchoolSchoolMETXID>
<!-- Kaltura wants the date to be in xs:long format, that is, seconds from the epoch -->
<METXVideoSubmissionCaptureDate>1344440267</METXVideoSubmissionCaptureDate>
</metadata>
</xmlData>
</customData>
</customDataItems>
</item>
</channel>
</mrss>
</code></pre>
<br />
And that's it.Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-15687900047244936182012-09-19T07:30:00.000-04:002012-09-19T07:30:00.266-04:00An Inconvenient OutageSome of you probably noticed that we had a rough weekend with the web site. We first saw trouble around 4pm EDT on Saturday. After some trouble shooting and investigation left us unsure of the root cause, we failed things over to our replica around 5pm EDT. We then ran off the replica over night and through the following morning.<br />
<br />
The big breakthrough came at 1:30pm or so Sunday when we isolated the cause, and then it took only a few minutes to correct the problem, test the solution, and finally roll service back to the production site. As with any longer outage this one pointed out a bunch of small, but important, changes to make in procedures and documentation.<br />
<br />
My apologies if you happened to be using our web site late afternoon on Saturday; the was certainly the roughest time.Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-75776153709487931762012-09-17T06:30:00.000-04:002012-09-17T14:32:43.229-04:00Introducing ICPSR's Virtual Data Enclave (VDE)The ICPSR Virtual Data Enclave (VDE) is a secure, virtual environment in which a researcher can analyze sensitive data, create research products, and then take possession of those products and analysis. And while he VDE is not a substitute for a physical enclave and the types of security protocols it facilitates, the VDE is very much a potential substitute for the traditional practice of distributing confidential data via removable media, such as CD-ROMs.<br />
<br />
The VDE uses much of the same technology that ICPSR uses internally for its Secure Data management Environment (SDE) which we have <a href="http://techaticpsr.blogspot.com/2011/11/icpsrs-secure-data-environment-overview.html" target="_blank">described</a> <a href="http://techaticpsr.blogspot.com/2011/08/icpsrs-secure-data-environment-sde_08.html" target="_blank">a</a> <a href="http://techaticpsr.blogspot.com/2011/07/icpsrs-secure-data-environment-sde_27.html" target="_blank">few</a> <a href="http://techaticpsr.blogspot.com/2011/07/icpsrs-secure-data-environment-sde.html" target="_blank">times</a>. In brief, we use a virtual desktop environment that is operated by the University of Michigan's central IT shop and connect it to what we call our Private Network Attached Storage (NAS) appliance. Both the virtual desktop and NAS are behind a firewall, and we use the firewall and Windows group policies to restrict what actions one pay perform. Download? Nope. Cut-and-paste between the virtual desktop and the real desktop? Uh uh. Capture screenshots by taking a picture of your monitor? Well, ......<br />
<br />
The virtual environment keeps sensitive datasets under lock and key at ICPSR, but makes it available to researchers. The environment contains the usual array of applications used in the social sciences (but no email!), exactly the same sort of stuff we might set up for a visiting scholar or OR.<br />
<br />
The researcher accesses the environment through a small, easy-to-download and -install client based on VMware View Client. Authentication takes place using standard University of Michigan credentials which we (ICPSR) and others at UMich can issue to "friends." Access between the real desktop and the virtual desktop is encrypted, and we are in the process of adding IPSEC encryption between the virtual desktop and the NAS. (This latter traffic passes over UMich's data backbone, and access to those routers is limited to UMich central IT network engineers.)<br />
<br />
The virtual machine is completely ephemeral and can be wiped after each use. Any intermediate research or results are stored on the ICPSR NAS. Our NAS is backed up weekly, and tapes are cycled off-site quarterly. Once the research has been completed ICPSR retains a "just in case you need it" snapshot for up to three years.Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-68285501777692929572012-09-14T06:30:00.000-04:002012-09-17T10:55:07.489-04:00Setting up Kaltura - part IVWe have been working on getting a Kaltura Drop Folder set up. A Drop Folder is a mechanism where an organization spools content to be ingested in a fixed location, and Kaltura polls the location, watching for content to ingest.<br />
<br />
In our case it has taken about a month to get the Drop Folder configured, and much of this delay is preventable if you avoid the same pitfalls we did. So in the spirit of giving back to the community, here are seven things to know when setting up a Drop Folder.<br />
<br />
<ol>
<li>Host the Drop Folder yourself, do not host it at Kaltura.</li>
<li>Set up an account and a password on the machine, and share them with Kaltura. To keep things very simple I created an account called 'kaltura'.</li>
<li>Create a subdirectory under the kaltura user's home directory that will actually contain the content to be ingested. To keep things very simple I used the name 'dropfolder'</li>
<li>Make sure that the kaltura user owns the Drop Folder directory, and that its access controls grant appropriate rights to other users that may need to ingest content</li>
<li>Tell Kaltura the name of the machine. To keep things very simple I created a DNS CNAME record, kaltura.icpsr.umich.edu, that points to the right machine.</li>
<li>Be sure you have ssh installed and running on port 22 on the machine. If you normally do not run ssh on port 22 (we don't), don't forget to open a hole in your firewall so that Kaltura machines can reach the Drop Folder.</li>
<li>Tell Kaltura to use sftp and port 22 to connect to your Drop Folder. Do not try to use a port other than 22.</li>
</ol>
<div>
To re-cap the values we used:</div>
<div>
<ul>
<li><span style="font-family: 'Courier New', Courier, monospace;">Host: kaltura.icpsr.umich.edu</span></li>
<li><span style="font-family: 'Courier New', Courier, monospace;">Protocol: sftp </span></li>
<li><span style="font-family: 'Courier New', Courier, monospace;">Port: TCP 22</span></li>
<li><span style="font-family: 'Courier New', Courier, monospace;">Login: kaltura</span></li>
<li><span style="font-family: 'Courier New', Courier, monospace;">Password: XXXXXXXX</span></li>
<li><span style="font-family: 'Courier New', Courier, monospace;">Drop folder: dropfolder </span></li>
<li><span style="font-family: 'Courier New', Courier, monospace;">Drop folder UID:GID: kaltura:met </span></li>
<li><span style="font-family: 'Courier New', Courier, monospace;">Drop folder mode: 2775</span></li>
</ul>
</div>
<div>
(We have a big video project called MET, and automated jobs running with the 'met' GID will need write access to the drop folder.)</div>
<div>
<br /></div>
<div>
You may be tempted to suggest using ssh keys or non-standard ports for ssh. Fight those temptations.</div>
<div>
<br /></div>
<div>
Kaltura will offer to auto-delete content once it has been ingested. Accept that offer.</div>
<div>
<br /></div>
<div>
Know that when you delete items from the Drop Folder status window in your Kaltura KMC it will also delete them from the Drop Folder. This is not obvious, but turns out to be useful.</div>
<div>
<br /></div>
<div>
Now all you need are automated jobs that place content and Kaltura-style Media RSS XML into the Drop Folder. Kaltura has some nice examples on-line, but they are somewhat trivial. We'll post some more complex, real-world examples next week.</div>
Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com2tag:blogger.com,1999:blog-4772431721558153311.post-30325384512233079972012-09-07T05:00:00.000-04:002012-09-07T05:00:01.535-04:00August 2012 depositsLight month for deposits:<br />
<br />
<table align="center" border="2">
<tbody>
<tr><td># of files</td><td># of deposits</td><td>File format</td></tr>
<tr><td>56</td><td>30</td><td>application/msword</td></tr>
<tr><td>69</td><td>26</td><td>application/pdf</td></tr>
<tr><td>9</td><td>8</td><td>application/vnd.ms-excel</td></tr>
<tr><td>2</td><td>1</td><td>application/vnd.wordperfect</td></tr>
<tr><td>4</td><td>1</td><td>application/x-dosexec</td></tr>
<tr><td>24</td><td>7</td><td>application/x-sas</td></tr>
<tr><td>47</td><td>21</td><td>application/x-spss</td></tr>
<tr><td>7</td><td>2</td><td>application/x-stata</td></tr>
<tr><td>26</td><td>2</td><td>application/x-zip</td></tr>
<tr><td>1</td><td>1</td><td>image/gif</td></tr>
<tr><td>18</td><td>4</td><td>image/jpeg</td></tr>
<tr><td>1</td><td>1</td><td>message/rfc8220117bit</td></tr>
<tr><td>3</td><td>3</td><td>text/plain; charset=iso-8859-1</td></tr>
<tr><td>17</td><td>10</td><td>text/plain; charset=unknown</td></tr>
<tr><td>32</td><td>9</td><td>text/plain; charset=us-ascii</td></tr>
<tr><td>3</td><td>2</td><td>text/plain; charset=utf-8</td></tr>
<tr><td>39</td><td>7</td><td>text/rtf</td></tr>
<tr><td>1</td><td>1</td><td>text/x-mail; charset=iso-8859-1</td></tr>
<tr><td>4</td><td>2</td><td>text/x-mail; charset=unknown</td></tr>
<tr><td>2</td><td>2</td><td>text/x-mail; charset=us-ascii</td></tr>
<tr><td>1</td><td>1</td><td>text/xml</td></tr>
</tbody></table>
<br />
Just the usual stuff, but in pretty low quantities.Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0tag:blogger.com,1999:blog-4772431721558153311.post-62413667389561621252012-09-05T06:00:00.000-04:002012-09-05T06:00:09.514-04:00ICPSR web availability - August 2012<div class="separator" style="clear: both; text-align: center;">
</div>
<div style="margin-left: 1em; margin-right: 1em;">
<img height="120" src="https://docs.google.com/a/umich.edu/spreadsheet/oimg?key=0AgUA8KteqtijdGx4cXl5UHl4M2Vmdi1Rc1Bya2NzZXc&oid=4&zx=qo68t4w0kt2w" width="400" />
</div>
<br />
<br />
August was not our best month.<br />
<br />
We did a bit better than 99.3% uptime. Almost all of the downtime is due to a recurring, as-yet-unsolved problem we are having with our Oracle database platform. The primary symptom is that the database platform stops fielding queries for about 5-15 minutes, which disables our production web site. The platform does this about 30-45 minutes AFTER it has finished a full export using the Oracle datapump system.<br />
<br />
Because our existing Oracle hardware is old and has a relatively slow disk I/O system, we're going to try to solve this problem by throwing hardware at it. For well under $10k we can replace our five-year-old hardware with something much newer. Goodbye RAID-5 SCSI, hello SSD.<br />
<br />
<br />Bryan Beecherhttp://www.blogger.com/profile/18073369761415410262noreply@blogger.com0