UTS eResearch

De-identifying survey data

2020-12-17T00:00:00+11:00

Researchers collecting data about individuals have various obligations to meet if they wish to share or publish the data. They may be required to seek written consent from participants to share their data, or ensure that it is de-identified to a point where it can’t be re-identified. The following gives two case studies of occasions where data custodians thought that they had adequately de-identified data, but close examination found that re-identification may be possible.

Defining personal information

Under the NSW Privacy and Personal Information Protection Act, personal information is

“any information or opinion (including information or an opinion forming part of a database and whether or not recorded in a material form) about an individual whose identity is apparent or can reasonably be ascertained from the information or opinion”.

Personal information can include:

Name, address, email address, phone number, date of birth, photographs, voice and video recordings, biometric information, IP address, location information from a mobile device, and government identifiers such as tax file number, Medicare number and driver’s licence number.

Sensitive personal information includes information relating to an individual’s ethnic or racial origin, political opinions, religious or philosophical beliefs, trade union membership or sexual activities. The Australian Privacy Act recognises sensitive personal information as a special class of personal information.

The NSW Health Records Information Protection Act defines health information as personal information that is about an individual’s physical or mental health, health services they have received, genetic information, healthcare identifiers, and information collected in the course of providing health care or donating body parts or organs.

De-identification

De-identifying data is not a straightforward process and researchers must take a number of factors into consideration prior to publishing data. As well as direct identifiers within the dataset itself, contextual information may make it possible to identify individuals. Thus, even if researchers have removed these identifiers, the dataset may not be suitable for publication if the participants consented to sharing of de-identified data. Refer to the de-identification pages from the privacy office for further guidance.

Case study: External data sources

In 2016 the Australian Department of Health published a dataset of Medicare and PBS data that researchers at the University of Melbourne were able to demonstrate had not been adequately de-identified. Cases in which they were able to re-identify data included cross-referencing surgery information with media reports of prominent individuals’ surgery, and using the dates that a woman gave birth, to identify individuals. Additionally, the researchers found that they could use the provider numbers in the dataset to identify multiple patients of the same medical practice. The Australian Privacy Commissioner found that the Department of Health had committed a privacy breach in publishing the dataset. The important lesson in this case is that an individual need not be identifiable to any casual observer but that someone with additional contextual information could identify an individual.

Case study: A survey of cancer patients

Researchers from UTS sought to publish responses to a survey of cancer patients about their experiences. All survey respondents provided information about the type of cancer they had had, when it had been diagnosed, how it had been treated, and their current status. In addition, respondents were given a number of opportunities to provide contextual comments throughout the survey. Close reading of these comments identified several instances of information that could, in context with other information identify respondents included:

Contact details: One individual included their email address in their response. Email addresses, along with telephone numbers, are generally considered to be personal information.

Dates of military service: The dates of someone’s tenure of a job, including military service, can be used in combination with other data to identify them.

Treatment centre: Information about the specific centre that an individual attended, alongside the information about their cancer type, dates, and treatment could allow identification.

Participation in a support program: Several members of a sporting team for cancer survivors mentioned joining this team, making them potentially identifiable to each other. Several other respondents mentioned a specific support or exercise program that they had accessed. Depending on the size of the group or program, this may make them identifiable to other participants.

Lessons

Researchers need to be alert to the possibility that participants in a study may be identifiable from a dataset even if direct identifiers have been removed. Factors that may make participants identifiable include membership of a small population (e.g., having a rare disease, being a member of a small team in an organisation), or being an outlier in a study (e.g., significantly older than most of the cohort).

Researchers need to take into consideration the possibility that external information may make an individual identifiable. For example, an athlete whose knee reconstruction was reported in the media may be identifiable in a dataset of knee surgery patients. Members of the study may also be able to identify each other from details in the data, for example as having attended the same program.

Researchers seeking to publish survey data should pay particular attention to free-text responses when preparing their dataset as participants may include information that could identify them, either directly or in context with other data.

An open, composable standards–based research eResearch platform: Arkisto

2020-11-23T00:00:00+11:00

This is a talk delivered in recorded format by Peter Sefton, Nick Thieberger, Marco La Rosa and Mike Lynch at eResearch Australasia 2020.

Research data from all disciplines has interest and value that extends beyond funding cycles and must be managed and preserved for the long term. However, much of the effort in eResearch goes into building systems which provide functionality and services that operate on data but which actually put data at risk. For instance, loading data into a particular tool often means that the data is not be easily retrievable if that tool or service cannot be sustained. At worst, the data is lost.

In this presentation we will introduce the standards based Arkisto platform and show a number of examples from multiple disciplines of current Arkisto deployments, including an institutional Research Data Portal, a snapshot of the Expert Nation history project, crowd-sourced data from historical criminology, and the Pacific And Regional Archive for Digital Sources in Endangered Cultures (PARADISEC).

Across the sector we build services that operate on data but which actually put data at risk.

The Arkisto (https://arkisto-platform.github.io/why/) approach is to work with a set of standards which make data available for long term access. The closest emoji I could find to represent standards was this “standard poodle”. Previously I used a toothbrush - on the basis that “standards are like toothbrushes, everyone wants to use their own”

🐩#1: Oxford Common File Layout

' title='4' border='1' width='85%'/>

The first of the two core standards is the Oxford Common File Layout (OCFL) to organize data in a repository as a set of files. This approach is scalable indefinitely, and reduces the risk that data will be locked up in monolithic systems.

This diagram by Mike Lynch shows a series of different sized collections of data, each with a label. The labels (manifests) in this case are purely about data integrity - and contain checksums. The bundles of data are the next level up as we move on to look at Standard number 2.

🐩#2: Research Object Crate

' title='5' border='1' width='85%'/>

RO-Crate is the standard Arkisto uses for packaging and describing data sets. It is based on other standards:

JSON-LD - an encoding scheme for linked-data using the universally accepted JSON format (Javascript Object Notation) to encode information about Data Entities such as files and folders and Contextual Entities such as people, places, instruments, data licenses. Files can be described in detail.

Schema.org is used as the main ontology for classes and properties - it has coverage for all the basic Who What Where style metadata and is used by Google’s dataset search and a number of other projects. There are a few terms from other ontologies where Schema.org does not have coverage.

RO-Crates may also have an HTML human readable summary of data. If you find a stray crate in your downloads folder it is easy to click on the HTML file and get a summary of what’s inside - they can also be hosted on the web using a plain-old webserver.

' title='6' border='1' width='85%'/>

This is a screenshot of an RO-Crate in the UTS data portal. We are looking at its HTML summary.

' title='7' border='1' width='85%'/>

With extensive metadata.

A growing set of Arkisto-compatible software tools allow data ingest into repositories, and the creation of data discovery portals that connect data to analytical, visualisation and computing tools.

OCFL Spec: https://ocfl.io/
Research Object Crate (RO-Crate) Spec: http://www.researchobject.org/ro-crate
UTS: https://github.com/UTS-eResearch/ro-crate-js
UTS OCFL JS Implementation: https://github.com/uts-eresearch/ocfl-js
UTS RO Crate / SOLR portal: https://github.com/uts-eresearch/oni-express
UTS Describo: https://github.com/UTS-eResearch/describo
UTS Describo Data Packs: https://github.com/UTS-eResearch/describo-data-packs
CoEDL OCFL JS implementation: https://github.com/CoEDL/ocfl-js
CoEDL Modern PARADISEC: https://github.com/CoEDL/modpdsc
CoEDL OCFL tools: https://github.com/CoEDL/ocfl-tools

' title='9' border='1' width='85%'/>

One important tool is Describo, a desktop (and soon to be online) tool for describing data using the RO-Crate standard. It creates linked-data descriptions that can describe a dataset at the top level, and also individual files or variables inside files.

There are two projects working on the online version of Describo - one at UTS and one led by CERN working with the European National Research Networks.

Describo can be configured for use in specific domains, for example in cultural archives like PARADISEC. This slide shows how users can create entities and link them, and select from pre-defined data loaded in as part of the profile.

' title='11' border='1' width='85%'/>

Arkisto currently has two data discovery tools that index the contents of an OCFL repository so humans and machines can discover data and connect to analytical, visualisation and computing tools. This is Michael Lynch’s diagram showing, from the left, how data can be “delivered” to a repository via standard tools (such as rsync) over SSH.

An indexing process uses Solr (or another index like Elasticsearch) builds an index of RO-Crate metadata that can be then used for search and faceted browsing over data. There is a user on the right, requesting access to a dataset and a security guard checking her credentials - the user has the rights to see datasets with a license - * - so in this case the system can serve the content to her.

Here is an example from the PARADISEC indexer. As per the Arkisto appraoach, the PARADISEC site is data driven - objects are stored on disk in OCFL using RO-Crate to describe each research object. Indexing tools walk the OCFL filesystem looking for RO-Crates then, using the crate metadata in addition to the OCFL inventory metadata, construct appropriate indexes into the content. In this example we can see version 1 of this item and the metadata we get from the OCFL inventory.

Expert Nation Snapshot

' title='13' border='1' width='85%'/>

This is an example of a faceted search interface constructed using the Oni portal tool developed at UTS. This image shows a data-export from the Expert Nation https://expertnation.org/ project (tagline “Universities, War and 1920s & 30s Australia”) led by Associate Professor Tamson Pietsch. Professor Pietsch asked us to create an archival snapshot of the state of the dataset to support a book. We are working with Pietsch’s team to configure the portal to be useful in exporting the data. The SectorName facet is particularly important; it shows that the health sector was by far the biggest employer of returned service people.

' title='14' border='1' width='85%'/>

Here is another dataset - this time we are looking at an RO-Crate sitting on a plain old web site (not a search portal). This is a screenshot of a map with a time-window function showing where one Laura Adams was convicted of 42 offences between 1918 and 1942. The power of the Arkisto platform, based on Standards, is that adding this kind of functionality to other collections with geographical features in it is a matter of writing a few simple bits of code. The component can be re-used because both the data and metadata use the RO-Crate standard (which in turn is built on other standards). The data in this demonstrator came from Alana Piper’s Criminal Characters project.

' title='15' border='1' width='85%'/>

This is a screenshot of geographical data about a single offender’s sentences that has been exported in to the Time Layered Cultural Map.

We are working on making this an automated service so that any Arkisto portal can be configured to display relevant geo-data and also to be able to export it for analysis to other tools via APIs including at large scale.

The researcher, Dr Alana Piper says:

Analytical possibilities here would be uploading all offenders in bulk and comparing the 'range' results to determine what types of offences or other factors are associated with higher/lower levels of mobility.

Modern PARADISEC

' title='16' border='1' width='85%'/>

A modern catalog driven from OCFL and RO-Crate. This is the landing page built from the content indexed into elastic search. We can see the number of collections, items, contributors and universities at a glance. There are controls for jumping to a specific item or collection and a simple auto-complete search for quickly finding known content. The bottom half is a dynamic list of the most recently updated items.

PARADISEC has viewers for various content types: video and audio with time aligned transcriptions, image set viewers, and document viewers (xml, pdf and microsoft formats). We are working on making these viewers available across Arkisto sites by having a standard set of hooks for adding viewer plugins to a site as needed.

PARADISE has advanced search and deep indexing into transcriptions with the ability to play segments directly from the search interface.

' title='19' border='1' width='85%'/>

This is another Arkisto based website - it’s a confidential, access-controlled database of successful grant applications built using an OCFL repository and RO-Crate objects and presented by the Oni portal.

' title='20' border='1' width='85%'/>

The Arkisto website has a growing list of use cases for different data pipelines - here’s a sketch of the architecture we’re working on for Associate Professor Shauna Murray’s group at UTS - managing data from a sensor network in estuaries along the NSW coast.

See the Use Cases page for more.

https://arkisto-platform.github.io

https://arkisto-platform.github.io ' title='21' border='1' width='85%'/>

To conclude:

Arkisto is a flexible research platform which can be used to assemble a variety of data pipelines, for a variety of disciplines.

The emphasis is on FIRST keeping data safe and re-usable by storing and describing it using standards, so that in the absence of budget and resources to maintain complex virtual labs the data are still available for re-use. We THEN use our growing set of interoperable tools to build data hubs with re-usable data viewer plugins and standards-based interoperable analytical services.

There are active projects underway at the University of Melbourne and University of Technology Sydney across a wide range of disciplines and we are seeking funding to enhance the platform.

eResearch Australasia 2019 trip report

2019-11-06T00:00:00+11:00

By Mike Lynch and Peter Sefton

Mike Lynch and Peter Sefton attended the 2019 eResearch Australasia conference in Brisbane from 22-24 October 2019, where we presented a few things - and a pre-conference summit on the 21st held by the Australian Research Data Commons, where Mike presented our report from our small discovery project on scalable repository technology. UTS paid for the trip.

What we presented - our work on Simple Scalable Research Data Repositories

We've posted fleshed-out versions of our conference papers as usual. Mike presented a short version of ARDC funded work on data repositories at both the the summit and the conference and Peter had also put in an abstract for longer version which is less technically focussed and gives more of the context for why this work is important.

Peter presented an update on our ongoing work on describing and packaging research data - this time focussing on the new merged standard Research Object Crate(RO-Crate) and looking at what's coming next.

Diversity breakfast

I (Mike) went to the breakfast given in honour of the late Dr Jacky Pallas, a senior figure in the eResearch community who had given a keynote at last year's eResearch Australasia

The speaker was Dr Toni Collis, a research software engineer and director of Women in High Performance Computing, on how lack of diversity is damaging your research, making the point that diverse research and support teams can be demonstrated to be more effective in terms of performance, and the importance of equity, diversity and inclusivity in attracting and retaining talent.

Research as a primary function of Electronic Health Records

Prof Nikolajz Zeps' presentation was one of a couple of talks which sold themselves as being provocative, in that they were arguing for a loosening of traditional, restrictive health care ethics and consent practices so that data could be made more readily available for research. He made the point that data sharing is very difficult in the Australian healthcare system not just because of ethical restrictions but because the system is so fragmented. He argued for an integration of research and clinical data consent and management practices, which would allow the information flows required for medical research to be used for the health care system itself to better monitor the effectiveness of treatments and patient outcomes. Moving the consent process from one of research ethics to clinical ethics can make things simpler, in terms of administration.

(I was less impressed by the other provocative keynote in which the speaker said "no-one ever died as the result of a health-care data breach", which I thought was a bit of posturing, even though it got applause from some of the audience.)

Notable presentations

Galaxy Australia and the Australian Bioinformatic Commons

These two presentations were part of the bioinformatics stream, about the Australian node of the global Galaxy workflow and computational platform, and the Australian BioCommons Pathfinder Project.

9 Reproducible Research Things

Amanda Miotto from Griffith University's eResearch team presented on running workshops to introduce researchers to good reproducibility practice which focussed on immediate benefits, like safeguarding a research team against individual members leaving or falling ill, and practical steps.

The workshop materials are available on GitHub

Physiome Journal

An interesting presentation by Karin Lundengård of the Auckland Bioengineering Institute about Physiome Journal, which will publish validated and reproducible mathematical models of physiological processes. The models will also be made available as Jupyter notebooks and shared on the Gigantum platform.

HASS BOF

A lot of what we spoke about in this BOF, which was chaired by Ingrid Mason, echoed what I (Mike) had heard the week before at a Big Data for the Digital Humanities symposium in Canberra which was organised by the ARDC and AARNet (and which I should write a blog post about).

The challenges faced by digital humanities researchers and support staff mirror one another - researchers are unsure of the right way to engage with technical staff and vice versa, and good collaboration is too labour-intensive to be sustainable if it's going to be spread beyond a minority of researchers who are already linked in to support networks.

This gave me a kind of wistful feeling about an earlier keynote from Dell about machine learning in medical science, because the speaker was very enthusiastic about moving HPC tools out of the realm where researchers needed to become technology experts to use them at all, into something more like commodity software. Although there are some areas of the humanities where this sort of thing is starting to happen - transcription is one which came up both in Canberra and Brisbane.

Data Discovery

I (Peter) chaired a session on Data Discovery with a couple of lead-in talks that outlined what's going on in the world of generic research data discovery, leading into a discussion.

From our viewpoint at UTS it was useful to get confirmation that discovery services are converging on using Schema.org for high-level description of data sets, for indexing by other services. Which is good, because that's the horse we bet on at UTS. It's being used by both Research Data Australia (RDA) and the new player Google dataset search (that's run by a tiny team apparently, but it will have a huge impact on how everyone has to structure their metadata).

Amir Aryani (Swinburne) and Melroy Almeida (Australian Access Federation) presented on ORCID graph looking at collaboration networks. This is testament to the power of using strong, URI-based identifiers, once you start doing that, metadata changes from an un-reliable soup of differently spelled ambiguous names, to something you can do real analytics on.

Adrian Burton (ARDC, ex ANDS) has been dealing with metadata a long time - he put it that the Schema.org approach had won, and suggested that this might be a bit of a loss. The RIF-CS standard that ANDS inherited and built RDA around had a entity based model, with Collections, Parties, Activities and services (based on ISO 2146) rather than simple flat name-value metadata. I agree that the entity model was a strength of RIF-CS But actually, for those that want to convey rich context about data, then schema.org with linked data can do everything RIF-CS can, more elegantly and with more detail. See the work we've been doing on RO-Crate which takes things to a deeper level with descriptions of files and (soon) even variables inside files including provenance chains (what people and equipment did to make those files from observations, or other files).

The leaders of that session have done a follow up survey, so I think they'll be putting out more info soon.

Preservation

The RO-Crate talk I gave (Peter here) was in a stream on Digital Preservation and data packaging.

Erin Gallant and Gavin Kennedy, also from AARNet about digital preservation.

I was talking, not taking notes, but we discussed what the research community and cultural collections folks can learn from each other - actually I think we made some of the same mistakes, both the eResearch community and GLAM sector invested in big silos which ended up not just storing data, but making it difficult to move, re-use. To labour the metaphor a bit, silos have small holes in the bottom, so getting data in and out is slow.

Mike's diagram of an OCFL Repository shows an alternative approach - instead of putting data in a container with constricted ingress and egress, lay it all out in the open. I'm not an expert in preservation systems, but I do know that that's the approach taken by the open source Archivematica preservation system (Note: I've done a bit of work for Artefactual systems which looks after it), it works as an application that sits beside a set of files on disk - if needed you can use the grandparent of all APIs, that is file operations to fetch data. All of our talks we gave, linked above, were about this idea in one way or another.

Trusted Repository certification - one for the UTS Roadmap

I (Peter again) attended a session MCed by Richard Ferrers from ARDC with contributions from people from a range of institutions and repositories who are part of an ARDC community of practice.

They talked about the Core Trust Seal repository certification program - and the process of getting certified.

Here's some background on CTS:

Core Certification and its Benefits

Nowadays certification standards are available at different levels, from a core level to extended and formal levels. Even at the core level, certification offers many benefits to a repository and its stakeholders. Core certification involves a minimally intensive process whereby data repositories supply evidence that they are sustainable and trustworthy. A repository first conducts an internal self-assessment, which is then reviewed by community peers. Such assessments help data communities—producers, repositories, and consumers—to improve the quality and transparency of their processes, and to increase awareness of and compliance with established standards. This community approach guarantees an inclusive atmosphere in which the candidate repository and the reviewers closely interact.

In addition to external benefits, such as building stakeholder confidence, enhancing the reputation of the repository, and demonstrating that the repository is following good practices, core certification provides a number of internal benefits to a repository. Specifically, core certification offers a benchmark for comparison and helps to determine the strengths and weaknesses of a repository.

Right now at UTS we're in the process of making a new Digital Strategy, aligned the UTS 2027 Strategy - one of the core goals (which are still evolving so we can't link to them just yet) is to have trusted systems. CTS would be a great way for the IT Department (that's us) to demonstrate to the organisation that we have the governance, technology and operational model in place to run a repository.

We're talking now about getting at least the first step (self certification) on the 2021 Roadmap - but before that, we'll see if we can join the community discussion and start planning.

This work is licensed under a Creative Commons Attribution 3.0 Australia License.

FAIR Simple Scalable Static Research Data Repository

2019-11-05T00:00:00+11:00

This presentation was given by Peter Sefton & Michael Lynch at the eResearch Australasia 2019 Conference in Brisbane, on the 24th of October 2019.

Welcome - we’re going to share this presentation. Peter/Petie will talk through the two major standards we’re building on, and Mike will talk about the software stack we ended up with.

' title='The project in a nutshell A static, file-based research data repository platform using open standards and off-the-shelf web technology OCFL – versioned file storage RO-Crate – dataset / object metadata Solr – index and discovery nginx – baked in access control

' border='1' width='85%'/>

This project is about building highly scalable research data repositories quickly, cheaply and above all sustainably by using Standards for organizing and describing data.

' title='

' border='1' width='85%'/>

We had a grant to continue our OCFL work from the Australian Research Data Commons. (I’ve used the new Research Organisation Registry (ROR) ID for ARDC, just because it’s new and you should all check out the ROR).

' title='

' border='1' width='85%'/>

OCFL Specifications

This Oxford Common File Layout (OCFL) specification describes an application-independent approach to the storage of digital information in a structured, transparent, and predictable manner. It is designed to promote long-term object management best practices within digital repositories. Specifically, the benefits of the OCFL include:

Completeness, so that a repository can be rebuilt from the files it stores

Parsability, both by humans and machines, to ensure content can be understood in the absence of original software

Robustness against errors, corruption, and migration between storage technologies

Versioning, so repositories can make changes to objects allowing their history to persist

Storage diversity, to ensure content can be stored on diverse storage infrastructures including conventional filesystems and cloud object stores

Source: https://ocfl.io

Here’s a screenshot of what an OCFL object looks like - it’s a series of versioned directories, each with a detailed inventory.

' title='

' border='1' width='85%'/>

One of the standards we are using is RO-Crate - for describing research data sets. I presented this at eResearch as well [TODO - link]

' title='

' border='1' width='85%'/>

This is an example of an RO-Crate showing that each Crate has a human-readable HTML view as well as a machine readable view.

🖥️ 👩🏾‍🔬 ' title='

🖥️ 👩🏾‍🔬 ' border='1' width='85%'/>

The two views (human and machine) of the data are equivalent - in fact the HTML version is generated from the JSON-LD version using a tool called CalcyteJS.

' title='

' border='1' width='85%'/>

This is a screenshot of work very much in progress - it’s a shows an example of the repository system working at the smallest scale, showing a single collection, “Farms to Freeways”; a social history project from Western Sydney, which we have exported into RO-Crate format as a demonstration. Each of the participants has been indexed for discovery. In a more deployment for a institutional repository, datasets would be indexed at the top level only. The point is to show that this software will be highly configurable.

' title='

' border='1' width='85%'/>

OCFL needs some explaining. I’ve had a couple of conversations with developers where it takes them a little while to get what it’s for.

' title='

' border='1' width='85%'/>

But they DO get it the standard is well designed.

' title='over to Mike ...

' border='1' width='85%'/>

' title='Off-the-shelf components

' border='1' width='85%'/>

Solr is an efficient search engine.

nginx is an industry-standard scalable web server, used by companies like DropBox and Netflix

Both are standard, open-source, easy to deploy and keep patched: unlike dedicated data repositories, which tend be fussy and make your server team swear.

https://my.repo/3eacb986d1.v4/PATH/TO/FILE.html

To a file in an ocfl repository:

/mnt/ocfl/3e/ac/b9/86/d1/v4/content/PATH/TO/FILE.html

' title='ocfl-nginx Resolves a URL:

https://my.repo/3eacb986d1.v4/PATH/TO/FILE.html

To a file in an ocfl repository:

/mnt/ocfl/3e/ac/b9/86/d1/v4/content/PATH/TO/FILE.html

' border='1' width='85%'/>

Mapping incoming URLs to the right file in the ocfl repository is straightforward and done with an extension in nginx’s minimal flavour of JavaScript. This slide simplifies things a bit: in real life we have URL ids which solr maps to OIDs and then to a pairtree path.

Todo: we want to use the Memento standard so that clients can request versioned rsources.

We are also looking at versioned DOIs pointing to versioned URLs and resources

' title='Code

' border='1' width='85%'/>

ocfl-js - a Node library for building and updating OCFL repositories ro-crate-js - a Node library for working with RO-Crates ocfl-nginx - an extension to nginx allowing it to serve versioned OCFL content a Docker image for ocfl-nginx solr-catalog - a Node library for indexing an OCFL repository into a Solr index data-portal - a single-page application for searching a Solr index

The codebase is in a lot of places but that’s consistent with the approach - they are all just components which we can deploy as we need them

The nginx extension is very small and would be easy to reimplement against another server

This is the most prototypical / primitive part of what we’ve got so far.

Licences on RO-Crate are indexed in the solr index. nginx authenticates web users, looks up which licences they can access, and applies access control to both search results and payloads.

At the moment, we’ve got a test server which doesn’t authenticate but which only serves datasets with a public licence and denies access to everything else.

' title='Access control at its most basic

' border='1' width='85%'/>

The screenshot on the left is a Solr query showing public and internal licences

The screenshot on the right is a basic web view of what nginx serves to an unauthenticated guest user - datasets with internal licenses aren’t shown

' title='Development strategy Agile development around well-designed data standards pays off Successful collaborations with PARADISEC and State Library of New South Wales, showing the feasibility and ease-of-use of both OCFL and RO-Crate It’s worth engaging with standards at the new/evolving stage, even if this requires a bit of running around to keep up

' border='1' width='85%'/>

Good data standards make incremental development much easier.

We were able to get real results in one- and two-day workshops with teams from PARADISEC and the State Library of New South Wales, both with large, structured digital humanities collections behind APIs.

Both the OCLF and RO-Crate standards are new and changing, but agile development means that it’s OK and even productive to keep pace with this and feed back into community consultation.

' title='

' border='1' width='85%'/>

In the last couple of months Marco La Rosa, an independent developer working for PARADISEC, has ported 10,000 data and collection items into RO-Crate format, AND built a portal which can display them. This means that ANY repository with a similar structure Items in Collections could easily re-use the code and the viewers for various file types.

' title='

' border='1' width='85%'/>

The Mitchell Collection - digitised public domain books with detailed metadata in METS and specialised OCR standards. We spend a day at the State Library and were able to successfully extract books into directories of JPEGs and metadata, package these using RO-Crate and start building an OCFL repository.

https://ocfl.io/ https://researchobject.github.io/ro-crate/ https://code.research.uts.edu.au/eresearch/solr-catalog/ https://github.com/UTS-eResearch/ocfl-js https://github.com/UTS-eResearch/ocfl-nginx Docker: mikelynch/nginx-ocfl

' title='Acknowledgements and links UTS: Moises Sacal Bonequi PARADISEC: Marco De La Rosa, Nick Thieberger State Library of New South Wales: Euwe Ermita

' border='1' width='85%'/>

This work is licensed under a Creative Commons Attribution 3.0 Australia License.

Meet RO-Crate

2019-11-05T00:00:00+11:00

By Peter Sefton

This presentation was given by Peter Sefton at the eResearch Australasia 2019 Conference in Brisbane, on the 24th of October 2019.

' title='Meet RO-Crate

' border='1' width='85%'/>

This presentation is part of a series of talks delivered here at eResearch Australasia - so it won’t go back over all of the detail already covered - see the introduction of datacrate in 2017 and and the 2018 update. The standard formerly known as DataCrate has been subsumed into a new standard called Research Object Crate - RO-Crate for short.

Eoghan Ó Carragáin https://orcid.org/0000-0001-8131-2150 (chair) Peter Sefton https://orcid.org/0000-0002-3545-944X (co-chair) Stian Soiland-Reyes https://orcid.org/0000-0001-9842-9718 (co-chair) Oscar Corcho https://orcid.org/0000-0002-9260-0753 Daniel Garijo https://orcid.org/0000-0003-0454-7145 Raul Palma https://orcid.org/0000-0003-4289-4922 Frederik Coppens https://orcid.org/0000-0001-6565-5145 Carole Goble https://orcid.org/0000-0003-1219-2137 José María Fernández https://orcid.org/0000-0002-4806-5140 Kyle Chard https://orcid.org/0000-0002-7370-4805 Jose Manuel Gomez-Perez https://orcid.org/0000-0002-5491-6431 Michael R Crusoe https://orcid.org/0000-0002-2961-9670 Ignacio Eguinoa https://orcid.org/0000-0002-6190-122X Nick Juty https://orcid.org/0000-0002-2036-8350 Kristi Holmes https://orcid.org/0000-0001-8420-5254 Jason A. Clark https://orcid.org/0000-0002-3588-6257 Salvador Capella-Gutierrez https://orcid.org/0000-0002-0309-604X Alasdair J. G. Gray https://orcid.org/0000-0002-5711-4872 Stuart Owen https://orcid.org/0000-0003-2130-0865 Alan R Williams https://orcid.org/0000-0003-3156-2105 ' title='

This is a recent snapshot of the makeup of the current RO-Crate team- compiled by Stian.

RO-Crate is a community effort to establish a lightweight approach to packaging research data with their metadata. It is based on schema.org annotations in JSON-LD, and aims to make best-practice in formal metadata description accessible and practical for use in a wider variety of situations, from an individual researcher working with a folder of data, to large data-intensive computational research environments.

' title='What is RO-Crate?

' border='1' width='85%'/>

The website says: RO-Crate is a community effort to establish a lightweight approach to packaging research data with their metadata. It is based on schema.org annotations in JSON-LD, and aims to make best-practice in formal metadata description accessible and practical for use in a wider variety of situations, from an individual researcher working with a folder of data, to large data-intensive computational research environments.

This is a timeline for the merging of the Research Object packaging work with DataCrate - again compiled by Stian. While our DataCrate work was driven by practical concerns and a desire to describe research data with high-quality metadata Research Object shared those concerns but with more of a focus on reproducibility and detailed provenance for research data.

' title='

' border='1' width='85%'/>

This is what an RO-Crate looks like if you open the HTML file that’s in the root directory (or you see one on the web).

' title='

' border='1' width='85%'/>

This is the home page for RO-Crate.

Where did RO-Crate come from? RO-Crate is the marriage of Research Objects with DataCrate. It aims to build on their respective strengths, but also to draw on lessons learned from those projects and similar research data packaging efforts. For more details, see background.

Who is it for?

The RO-Crate effort brings together practitioners from very different backgrounds, and with different motivations and use-cases. Among our core target users are: a) research engaged with computation and data-intensive, wokflow-driven analysis; b) digital repository managers and infrastructure providers; c) individual researchers looking for a straight-forward tool or how-to guide to “FAIRify” their data; d) data stewards supporting research projects in creating and curating datasets.

RO-Crate is a collaboration between people all over the world, but the Editors are from Cork, Manchester and Katoomba Version one of the standard will be out in by Summer. But which summer? Standard reference points are important. Standards are important.

Which brings us the benefits of Standards. Without this standardised date format chaos would reign. What if that date had been written 05/08 or 08/05 - someone might end up eating food from May in August, or worse, eating last August’s food in May.

Anyway, If you find a partner who’ll adopt the ISO 8601 data standard then ...

… you should marry them.

Like how we married the Research Object and DataCrate - we bonded over standardisation.

Let’s explore standards a bit more. Iif you see this in metadata - what does it mean?

Is it a name given to the resource? URI: https://www.dublincore.org/specifications/dublin-core/dcmi-terms/terms/title/

An honorific like Ms, or Dr? As it would be in the FOAF ontology.

Or a very specific meaning relating to job titles? As in Schema.org.

In RO-Crate - there’s an HTML page which ships with each dataset that allows you to browse the object in as much detail as the author described it and we are careful to avoid ambiguity by adding help links to each metadata term so you see the definition.

' title='

' border='1' width='85%'/>

Just wanted to shout out to ResearchGraph - led by Amir Aryani at Swinburne Uni - they are also using schema.org.

🖥️ 👩🏾‍🔬 ' title='

🖥️ 👩🏾‍🔬 ' border='1' width='85%'/>

RO-Crates ship with two files, a human readable one and a machine readable JSON file. The two views (human and machine) of the data are equivalent - in fact the HTML version is generated from the JSON-LD version, via the DataCrate nodejs library.

' title='

' border='1' width='85%'/>

And here’s an automatically generated diagram extracted from the sample DataCrate showing how two images were created. The first result was an image file taken by me (as an agent) using two instruments (my camera and lens), of a place (the object: Catalina park in Katoomba). A sepia toned version was the result of a CreateAction, with the instrument this time being the ImageMagick software. The DataCrate also contains information about that CreateAction such as the command used to do the conversion and the version of the software-as-instrument.

convert -sepia-tone 80% test_data/sample/pics/2017-06-11\ 12.56.14.jpg test_data/sample/pics/sepia_fence.jpg

This way of representing file provenance is Action-centred - the focus is on the action that creates a file, rather than the more usual metadata approach of having the file at the centre with properties for “Author” and the like. The action-based approach is MUCH more flexible as it can model the contribution of multiple agents and instruments separately at the expense of being somewhat counter-intuitive to those of us who are used to a library-card approach to metadata where the work is at the centre and has simple properties.

There was a question after this presentation about whether I had the arrows in this diagram pointing in the right direction. Yes, I do! The convention here is the standard way of representing a subject-predicate-object semantic triple with the subject as the source of the arrow, the predicate (in this case Schem.org property) as a label, and the pointy end pointing at the object.

' title='🐥

' border='1' width='85%'/>

What’s new / developing at the moment in the RO-Crate world? I will illustrate by looking at recent activity on our Github project.

' title='

' border='1' width='85%'/>

We’re working on ways to describe not just files, but the CONTENTS of files - using properties like variableMeasured.

' title='

' border='1' width='85%'/>

We have a way to describe a workflow

' title='

' border='1' width='85%'/>

and actions that can be performed on data such as firing up a computational environment to re-run the workflow.

' title='

' border='1' width='85%'/>

You too can add Use Cases like this one about software containers.

' title='

' border='1' width='85%'/>

Breakig news: In the last couple of months Marco La Rosa, an independent developer working for PARADISEC, has ported 10,000 data and collection items into RO-Crate format, AND built a portal which can display them. This means that ANY repository with a similar structure Items in Collections could easily re-use the code and the viewers for various file types.

http://45.113.232.73/paradisec.org.au/NT1/98007 ' title='

http://45.113.232.73/paradisec.org.au/NT1/98007 ' border='1' width='85%'/>

This shows an intralinear transcription where you can play various segments of a recording and see the transcription.

' title='

' border='1' width='85%'/>

The .eaf files in the previous example are produced using ELAN software. Marco has done the groundwork for a system that could work across multiple repositories and for stand-alone RO-Crates - the crate metadata describes the files, and what format they’re in, and the viewer which is an HTML page either served by a repository or possibly just off your hard disk, can use that information to load an appropriate viewer.

' title='

' border='1' width='85%'/>

RO-Crate will be released in version 1 in November 2019 - we were aiming for October, but missed that.

We will publish the parts that are well-tested and stable, and immediately start on a new version with bleeding-edge cases.

We want input from potential users, current and prospective implementers and help drafting new parts of the spec is welcome.

You can join the team

    <a rel="license" href="http://creativecommons.org/licenses/by/3.0/au/"><img alt="Creative Commons Licence" style="border-width:0" src="https://i.creativecommons.org/l/by/3.0/au/88x31.png" /></a><br />This work is licensed under a <a rel="license" href="http://creativecommons.org/licenses/by/3.0/au/">Creative Commons Attribution 3.0 Australia License</a>.

Publishing versioned datasets using OCFL and nginx

2019-11-05T00:00:00+11:00

ARDC funding - Data and Services Discovery projects - Institutional Role in a Data Commons

There are a lot of specialized repository applications, from small (Omeka) to large (Hydra, Fedora), all designed as special-purpose homes for datasets and metadata which provide APIs for getting things in and out.

Experience has shown that these solutions don’t scale. Eventually, an institution will have to store a dataset that’s too big either to get in or out, or to store, and will have to look at a workaround like putting the data on disk and pointing to it from a record in the repository.

The Oxford Common File Layout (OCFL) is a standard for laying out arbitrary collections of files as structured repositories. It grew out of a push from the repositories community for a repository structure that wasn’t locked in to a particular application, didn’t have scaling problems, and was easy to migrate to and from.

An OCFL repository is a collection of OCFL Objects, laid out according to a simple standard like PairTree. Objects are immutable and versioned. More details can be found at our FAIR Repositories presentation.

We’re using OCFL for our data publications repository, and will later use it for an internal data repository with access control.

RO-Crate is a standard for describing individual datasets which evolved from two previous standards, DataCrate and Research Objects. Datasets are described using JSON-LD, a standard for building linked data descriptions in JSON. The ontology is Schema.org, which is widely supported in industry, and used by Google’s new dataset search engine.

A “crated” dataset is a directory with an arbitrary file hierarchy inside it, and an RO-Crate JSON-LD document with contextual metadata (title, description, contributors, licences) and descriptions of some or all of the contents. An RO-Crate doesn’t have to describe every file inside it, as this would be impractical for some datasets.

An RO-Crate doesn’t even have to have any files other than the RO-Crate files. This allows the system to support metadata-only publication, where the actual data is only available on request.

An RO-Crate also contains an HTML document which is generated automatically from the JSON-LD. This provides a human-readable view of the metadata with links to the data payloads. It can be used locally or act as the landing page for an RO-Crate published on a web server.

The HTML document is rendered through some lightweight JavaScript, but will degrade gracefully for browsers or users who don’t support JavaScript.

OCFL and RO-Crate are the standards we’re using to lay out the repository. To deliver them, we’re using Solr, an efficient search engine, and nginx, a high-performance open-source web server.

Metadata from the RO-Crates is indexed into Solr, including licences representing lists of users who are allowed to access the dataset. Solr is also used to provide a discover interface via a lightweight single-page application.

A simple extension to nginx allows it to resolve incoming URLs to paths in the OCFL repository and serve the appropriate version of the RO-Crate metadata and payload files.

Before serving a file, the nginx handler looks up the Solr index and checks whether the authenticated user has authority to download it.

Nginx enforces licences on both the search results and payloads.

' title='Examples In production for UTS data publication repository PARADISEC – crosswalking to OCFL and RO-Crate State Library of NSW – Mitchell collection of public domain books

' border='1' width='85%'/>

The OCFL/RO-Crate and nginx stack is meant to accommodate datasets from a wide range of disciplines and sources, from small web uploads, existing repositories and data capture.

As part of the ARDC Data and Services Discovery project, we collaborated with Nick Thieberger and Marco La Rosa of PARADISEC and Euwe Ermita’s team at the State Library of NSW. Both of these institutions have rich digital humanities collections stored in custom repositories behind APIs, and in short day- or two-day workshops we were able to make substantial progress in crosswalking their datasets into OCFL.

This research/project is supported by the Australian Research Data Commons (ARDC). The ARDC is enabled by NCRIS. ' title='Links and acknowledgements https://ocfl.io/ https://researchobject.github.io/ro-crate/ https://github.com/UTS-eResearch/ocfl-nginx Docker: mikelynch/nginx-ocfl

This research/project is supported by the Australian Research Data Commons (ARDC). The ARDC is enabled by NCRIS. ' border='1' width='85%'/>

This work is licensed under a Creative Commons Attribution 3.0 Australia License.

eResearch Survey Report

2019-10-01T00:00:00+10:00

Overview

In order to know whether we are providing what researchers need and thereby to improve our services to researches, we rolled out a short anonymous survey* back in May 2018 that has been open to all UTS researchers, research assistants and HDR students.

We have received 41 responses so far. The two bar charts below show the distribution of their research positions and the distribution of their research fields respectively.

Left: Distribution of participants' research positions. Right: Distribution of participants' research fields.

What researchers say

In qualitative comments from the survey, we received feedback that:

“Hacky hour is the best thing about UTS. I would love to have more than one session per week if resources allow: Monday and Thursday for example. Sometimes when dealing with a coding problem what I can't solve it seems like Thursday is so far away. Thank you for your support so far.”

However, we did not meet the needs of all. One suggestion was to:

“Provide services at a time not in business hours for part time students that cannot come to workshops/support services in business hours.” There were also a few grumbles about our HPC/iHPC that are beyond our budget to fix although the staff who run them came in for high praise. Overall the tone was: “You guys do a great job. Keep it up.”

The most commonly voiced criticism was that eResearch services were difficult to locate:

“Information about what services were available, where to go to access these services, regular reviews and updates on these services would be helpful. Currently services like lab archives etc I found out about by lunch room chat.”

We have tried a variety of avenues of communication, such as workshops, Staff Connect, Service Connect, our website, UTS library website, faculty research mailing lists, GRS mailing lists, on-campus screens advertising Hacky Hour, Hacky Hour and even this blog. None of these methods will satisfy everyone. In fact, one comment read “PLEASE STOP SPAMMING US WITH YOUR EMAILS”, illustrating the frustration with which researchers face a deluge of emails.

Another participant noted that a solution would be:

“one pathway to lead to accessing advice rather than multiple lines of inquiry to get to Hacky hour as the place for expert sharing of advice in research design using Lime survey and CloudStor etc.”

eResearch is therefore excited to be involved in ResHub which, even if it begins simply as a one-stop directory to researcher services, will be meeting a widely-voiced need.

Usefulness and Delivery of eResearch Services

In general, the participants find many services useful for their research, and think the delivery of the services have been satisfactory.

Specifically, for each service we asked researchers to provide feedback on, the chart on the left shows to what extent they think the service is useful for their research, and the chart on the right shows how they think the service is delivered (N/A means they have not used the corresponding service yet).

Research data management services (e.g. Stash, Cloudstor, REDcap, eResearch Store, Gitlab, Omero, eNotebooks)

Left: Usefulness of Research data management services. Right: Delivery quality of Research data management services.

Other data services (e.g. data archiving, data publication, data visualisation, data arena)

Left: Usefulness of Other data services. Right: Delivery quality of Other data services.

Survey platform services (LimeSurvey, REDcap)

Left: Usefulness of Survey platform services. Right: Delivery quality of Survey platform services.

UTS HPC services (e.g. UTS-HPCC, FEIT ARClab)

Left: Usefulness of UTS HPC services. Right: Delivery quality of UTS HPC services.

External HPC services (AWS, Nectar, NCI, Intersect)

Left: Usefulness of External HPC services. Right: Delivery quality of External HPC services.

eResearch consulting services (e.g. grants/ethics advice, collaboration tools, research data consultation, data security)

Left: Usefulness of eResearch consulting services. Right: Delivery quality of eResearch consulting services.

Training services through Intersect (e.g. REDCap, R, Python, Excel)

Left: Usefulness of Intersect training services. Right: Delivery quality of Intersect training services.

Training through UTS library (e.g. Research Data Management, eNotebooks)

Left: Usefulness of Library training services. Right: Delivery quality of Library training services.

"Hacky Hour" researcher support

Left: Usefulness of Hacky Hour. Right: Delivery quality of Hacky Hour.

ResBaz Sydney 2017

Left: Usefulness of ResBaz Sydney 2017. Right: Delivery quality of ResBaz Sydney 2017.

*: Study data were collected and managed using REDCap, a secure, web-based software platform designed to support data capture for research studies.

Roadmap for Genomics Computing Workshop

2019-07-02T00:00:00+10:00

This Genomics Workshop was held at Sydney University on 3rd June 2019 Pascal Tampubolon and Mike Lake attended the event. Present were the NSW Chief Scientist and leaders of various Genomics institutes in Australia.

The University of Sydney are developing a 1-5 year infrastructure roadmap to enable excellence in genomics-based research and a strategy to extend this across other institutions in Australia. This workshop showcased some outstanding genomics research being done Australia. A panel discussion by leaders in the genomics and genomics infrastructure followed.

Common and major technical issues raised by attendees were: requirements for increased space on /scratch on HPCs and long compute times, problems with I/O bottlenecks in sharing data with collaborators, and the responsibilities and costs of storing genomics data. After an ARC grant has finished who stores the data and who pays for the data to be stored?

Common and major technical issues raised by attendees were: storage requirements for research data (compute and archival purposes), long HPC compute and queue times, I/O bottlenecks in sharing data with collaborators, and the responsibilities and costs of storing genomics data.

A number of institutions are "bursting into cloud" for compute and they want to move the compute to where the data is. Though no one suggested how this could be done. Some users of the Syd Uni HPC are looking to migrate pipelines to AWS.

The ongoing problem of data curation was also raised as well as the repoducibility of re-running analyses. Using PiGx for reproducible pipelines across different infrastructure was mentioned.

Tony Papenfuss from the Walter and Eliza Hall Institute mentioned that they were using more GPU cards for processing jobs and that purchasers of new sequencers are required to specify compute and data storage requirements prior to purchasing.

Jon Smillie from NCI told us about the use of Crypt4GH, an encrypted file format, from the Global Alliance for Genomics and Health (GA4GH).

Andrew Gilbert, General Manager, Bioplatforms Australia spoke about the newly formed Australian Bioinformatics Commons and specifically drew our attention to the "BioCommons Principles".

Attendance was valuable to us in order to meet other researchers and managers of genomics institutes, to be aware of shared problems and possible future strategies.

Mike Lake & Pascal Tampubolon
eResearch

DataCrate - a progress report on packaging research data for distribution via your repository

2019-07-01T00:00:00+10:00

' title='DataCrate: a progress report on packaging research data for distribution via your repository Peter Sefton University of Technology Sydney

' border='1' width='85%'/>

This is a talk that I delivered at Open Repositories 2019 in Hamburg Germany, reporting on developments in the DataCrate specification for research data description and packaging. The big news is that DataCrate is now part of a broader international effort known as RO-Crate. I spent several hours at the conference working with co-conspirators Stian Soiland-Reyes and Eoghan Ó Carragáin on the first draft of the new spec which we hope to unveil at eResearch Australasia 2019.

Eoghan, Stian and I ran a workshop at OR2019 for repository people to talk about the state of the art in Research Data Packaging, and collect use cases - we got lots of useful input from the workshop and the broader conference and had a chance to chat with people working on related standards such as the Oxford Common File Layout (OCFL) and Sword 3 - Neil Jeffries, Andrew Woods, Simeon Warner and my old mate Richard Jones amongst others.

I also presented work that Mike Lynch, Moises Sacal and I have been doing on OCFL.

My travel was funded by the University of Technology Sydney.

Peter Sefton

University of Technology Sydney, Australia

DataCrate is a specification for packaging research data for dissemination and reuse which has been presented at OR before as it developed to its current v1.0 status. This is an update on progress with the specification and tooling. The goals are of the specification are, (a) to maximise the utility of the data for researchers (including the original researchers' 'future selves') - given that a researcher has found a DataCrate package they should be able to tell what it is, how the data may be used and what all the files contain, (b) to enable discovery of the data by exposing metadata as widely as possible to both humans and machines and (c) to enable automated ingest into repositories or catalogues. DataCrate can express detailed information about which people, instruments and software were involved in capturing or creating data, where they did it and why, as well as how to cite a dataset. DataCrate draws on other standards (BagIt, JSON-LD, Schema.org) and is designed to be easy to implement.

When I proposed this update there had been some work going on to merge DataCrate with another standard - Research Object - but we didn’t know what form that would take. This presentation will now let the love-stuck cat out of the bag. I’m going to tell you a love story about two standards.

The story is about DataCrate - which is a Specification for describing and distributing data (any data).

And Research Object which is also about data packaging but with more emphasis on having a major impact on scholarly communications and practice, including distributing research data with code and workflows in the interests of making it more reproducible.

There’s going to be a marriage of these two standards.

But before we talk about the upcoming marriage of two specifications, let’s go back over what we’re trying to achieve. I will use examples from DataCrate but the same principles apply with Research Object.

Here’s a slide from the workshop we ran at the OR conference, setting out some of the functional requirements for a distribution format.

← CLICK HERE TO DOWNLOAD ' title='

← CLICK HERE TO DOWNLOAD ' border='1' width='85%'/>

Here’s a screenshot from a sample DataCrate. It shows the basic metadata for the dataset. This example is online, but the same view is available if you download it as a zip file.

← CLICK HERE ' title='

← CLICK HERE ' border='1' width='85%'/>

If you do download the zip then you’ll see these files - the payload data is/are in the /data directory and the other files are all metadata and fixity checking information. A human can open CATALOG.html and then click around a little website that describes the data, including information down to the file level and about the contents of files.

← Feed this to your computer ' title='

← Feed this to your computer ' border='1' width='85%'/>

There’s also machine readable data in JSON-LD format.

' title='

' border='1' width='85%'/>

The JSON-LD is designed to be easily consumed by programs that can do things with the data - and is compatible ith Google’s DataSet search.

🖥️ 👩🏾‍🔬 ' title='

🖥️ 👩🏾‍🔬 ' border='1' width='85%'/>

The two views (human and machine) of the data are equivalent - in fact the HTML version is generated from the JSON-LD version using a tool called CalcyteJS.

' title='

' border='1' width='85%'/>

Here’s a screenshot of an HTML page about one of the files in the sample dataset - including detailed EXIF technical metdata which from INSIDE the file.

' title='

' border='1' width='85%'/>

convert -sepia-tone 80% test_data/sample/pics/2017-06-11\ 12.56.14.jpg test_data/sample/pics/sepia_fence.jpg

' title='URIs as names for things

' border='1' width='85%'/>

Because DataCrate is based on JSON-LD, and linked data principles, each term used can have a link to its definition, eg: https://schema.org/CreateAction so DataCrates are self-documenting.

' title='But (🔧🔨🔩🔪🔬)ing is limited: JSON-LD → 🙅 ⛓💾 → 👶

' border='1' width='85%'/>

BUT. tools for humans to generate linked-data are under-developed.

JSON-LD tooling is limited to high-level transformations and there are no easily available libraries for Research Software Engineers to do simple stuff like traversing graphs or looking up context keys.

🚀⚗️ 👩🏽‍🚀👷‍🏗️🏢 🛐 ' title='⛓💾’s still too much like

🚀⚗️ 👩🏽‍🚀👷‍🏗️🏢 🛐 ' border='1' width='85%'/>

Linked data is still too much like rocket science because of all the architecture astronauts:

Joel Splosky:

These are the people I call Architecture Astronauts. It’s very hard to get them to write code or design programs, because they won’t stop thinking about Architecture. They’re astronauts because they are above the oxygen level, I don’t know how they’re breathing. They tend to work for really big companies that can afford to have lots of unproductive people with really advanced degrees that don’t contribute to the bottom line.

Also, as with many technologies, RDF can be a bit of a religious matter.

Speaking of relgion, back to the wedding of the decade ...

The new entity is called RO-Crate.

We are collecting use-cases in the github repo. We collected several at our OR2019 workshop, and the the repository is still open for business.

Members of the OR-Crate project (anyone can join by following the directions at the repo) are reviewing the spec - which is based on a set of examples - and we expect to have a simplified, clearer specification draft by the end of July, and to launch an Alpha version in October at eResearch Australasia (subject to getting presentation accepted :).

' title='What are we working on Guidance on how to generate schema.org based metadata for datasets: Who created it, what is the subject matter? what are these files? where was it made? What format is each file in? where is about, why was it made? (ie what funded project is it part of?) How does RO-Crate work with BagIt, OCFL, Zip et al How can I re-run this analysis? Via a workflow? How can I do stuff with this dataset?

' border='1' width='85%'/>

We’re working on merging the DataCrate simple-to-implement approach with the bigger vision of Research Object.

' title='

' border='1' width='85%'/>

Join us!

Implementation of a Research Data Repository using the Oxford Common File Layout standard at the University of Technology Sydney

2019-07-01T00:00:00+10:00

This is a presentation by Michael Lynch and Peter Sefton, delivered by Peter Sefton at Open Repositories 2019 in Hamburg. My travel was funded by the University of Technology Sydney.

' title='Implementation of a Research Data Repository using the Oxford Common File Layout standard at the University of Technology Sydney Michael Lynch, Peter Sefton University of Technology Sydney, Australia

' border='1' width='85%'/>

This presentation will discuss an implementation of the Oxford Common File Layout (OCFL) in an institutional research data repository at the University of Technology. We will describe our system in terms of the conference themes of Open and Sustainable and with reference to the needs and user experience of data depositors and users (many have large data, and or large numbers of files). OCFL, which is an approach to repository implementation based on static data, was developed to deal with a number of issues with “traditional” repository design, many of which are particularly acute when dealing with research data. We will cover how this meets our user and institutional needs and is a sustainable approach to managing data.

This was presented by Peter Sefton - so the “I” throughout is him.

This presentation was in a session about OCFL so we didn’t need to explain the standard in detail.

Here’s what they say on the OCFL site:

This Oxford Common File Layout (OCFL) specification describes an application-independent approach to the storage of digital information in a structured, transparent, and predictable manner. It is designed to promote long-term object management best practices within digital repositories.

Specifically, the benefits of the OCFL include:

Completeness, so that a repository can be rebuilt from the files it stores

Parsability, both by humans and machines, to ensure content can be understood in the absence of original software

Robustness against errors, corruption, and migration between storage technologies

Versioning, so repositories can make changes to objects allowing their history to persist

Storage diversity, to ensure content can be stored on diverse storage infrastructures including conventional filesystems and cloud object stores

' title='

' border='1' width='85%'/>

Also Moises Sacal worked with us on this.

" title='> repo = new Repository()

' border='1' width='85%'/>

Hello and welcome to this presentation about OCFL at UTS. Don’t be alarmed - yes there’s javascript code on the screen, but there will not be a test!

Repository { ocflVersion: '1.0', objectIdToPath: [Function] }

' title='> repo = new Repository()

Repository { ocflVersion: '1.0', objectIdToPath: [Function] }

' border='1' width='85%'/>

To me, being able to type npm install ocfl then instantiate a repository in an interactive shell is quite amazing - when I first worked with repositories, from about 2006, installing a repository was a big job, there were usually lots of prerequisites and installation (for example see this guide to installing UQ’s Fez Fedora repository could be very finicky, but then in those days the repositories were “full stack” services whereas OCFL is just about the heart and soul of a repository, the storage.

A confession.

And some philosophy: “What is a ‘repository’ anyway” ' title='What you’re going to get An explanation of OCFL ... ... from a UTS perspective.

A confession.

And some philosophy: “What is a ‘repository’ anyway” ' border='1' width='85%'/>

This talk is not just about OCFL, it reflects on the role of repositories and how we should view them in our infrastructure.

My first “Confession”.

Many, many years ago I was the technical manager for the RUBRIC project - Regional Universities Building Research Infrastructure Collaboratively. This project helped a group of about nine partner unis get their first publications repositories up and running and our team morphed into the the national repository support service for Australia. As part of this work we built a variety of migration tools to assist people in getting data from existing systems such as endnote, or ingesting MARC (library) metadata into a repository. To do this we used the DSpace archive format, a simple directory-plus-metadata file format that was not unlike OCFL.

It occurred to me at the time that we could build a simple static repository system that used something like the DSpace archive format, with separate ingest workflows, and build a portal using something like Apace Solr (though at that stage there really wasn’t anything else like Apache Solr).

I didn’t follow up on those thoughts, as I thought they were a bit heretical - the monolithic repository architecture was the orthodoxy, though I did flirt with this idea in this article about discovery portals and persistent identifiers. Anyway, it seems like those wrong thoughts are something I can talk about, now that we have OCFL.

' title='

' border='1' width='85%'/>

Confession 2

When I was working on the Australian Government funded NeCTAR Virtual Lab project, Alveo I suggested we start with a Hydra/Fedora repository as the ‘heart’ of the Virtual Lab, the repository component. That worked well in that we got up and running very fast with a discovery portal for data - but ran into problems when it came to data access - for example making Item Lists - data-sets - performed terribly because of the overhead inherent in the Hydra architecture.

I delivered a presentation at Open Repositories 2014, written with other Alveo staff, that explored some of the problems with using a “full stack” repository at scale.

Initially (and throughout the project) Steve Cassidy was suggesting a more OCFL-like way of loading data where it would be arranged on disk by some kind of curation human or machine orchestrated process and then indexed. Steve was right. I was wrong. I’d love to come back to the Alveo lab architecture for another go, maybe in a future round of ARDC investment?

' title='> repo.create("/Users/124411/everything")

' border='1' width='85%'/>

Like I said, creating a repository is this simple.

' title='What is a repository anyway?

' border='1' width='85%'/>

But what is a repository?

There are a variety of ways to look at a “repository” - all of these are facets of what a repository can be.

' title='🛒 📬 🐿 💽 🗃🗄🔭🔎🔬

' border='1' width='85%'/>

Repositories are no longer just places to put stuff (if they ever were) - they’re positively bristling with services. Over the 13 years of this conference, repositories have become shall we say baroque. And in some cases slow. They also tend to have problems when you present them with a terabyte file, or ten million 1 kilobyte files. OCFL is, we think, in part a reaction from our community to those realities.

' title='CON

' border='1' width='85%'/>

Using OCFL as the storage layer in a repository is a radical separation of con-

' title='CERNS

' border='1' width='85%'/>

-cerns.

' title='

' border='1' width='85%'/>

This is a view of our eResearch Architecture - Stash is our Data Management service, which is an implementation of ReDBox. The point of showing the diagram is not to go through it all in detail, but to show how complex the ecosystem of research data services can be - and where data archive / repository core sits. The OCFL components (the publication and archive file-systems) are shown in red, while Agents (the repository Adaptor that writes content from Research Workspace Services in a repository, the Public and Private data portales) that access OCFL are shown in blue.

' title='

' border='1' width='85%'/>

This is our work-in-progress discovery portal - this is not built in to the repository like it would be with, say DSpace, it’s a separate service that indexes an OCFL repository (in this case the public data).

' title='

' border='1' width='85%'/>

Here’s the same diagram stripped back a bit more ...

' title='

' border='1' width='85%'/>

And even more stripped back, to show only the core repository service (in red) and a couple of services that interact with it (in blue).

And here’s the actually repository part - the storage layer.

' title='

' border='1' width='85%'/>

The idea of OCFL is that there are services that all access this core layer but they’re disposable and/or interchangeable, and you just leave your precious data where it is, on disk (or some virtual disk-like view of whatever the storage solution de jour happens to be).

Here’s a screenshot of what an OCFL object looks like - it’s a series of versioned directories, each with a detailed inventory.

' title='

' border='1' width='85%'/>

No this slide is not about a zoological repository. It’s about the Elephants in the repository AKA the risks that come with an OCFL implementation.

The main issue in implementation is making sure that you have to THINK about the design of software to update the repository at the same time. You don’t want to end up with (a) corrupt files or (b) transactions that didn’t complete.
OCFL is also very new (it’s not at version 1 yet) so there’s some risk that the services we are hoping for won’t arrive - but given that our data are still safe on our own storage infrastructure and it possible to code-up an OCFL consuming application in a matter of days this is not a serious risk.

' title='

' border='1' width='85%'/>

OCFL needs some explaining. I’ve had a couple of conversations with developers where it takes them a little while to get what it’s for.

' title='

' border='1' width='85%'/>

But they DO get it.

Mike Lynch’s summary - this is a modern take on “how to I organise my stuff” - in this sense OCFL is like a framework for data - and we all use frameworks for code these days,

' title='

' border='1' width='85%'/>

We have some good news to announce - we have a grant to continue our OCFL work from the Australian Research Data Commons. (I’ve used the new Research Organisation Registry (ROR) ID for ARDC, just because it’s new and you should all check out the ROR).

We’re going to be demonstrating large-scale use of OCFL, with research objects described using RO-Crate metadata. (See also my presentation on DataCrate which introduces RO-Crate).

' title='🤔

' border='1' width='85%'/>

So what do we think?

Working with OCFL has been really great for the team at UTS - it’s well designed, and just when you think “hey, how do I do file-locking” you find that there’s a hint in the design (a /deposit directory in this case) that points towards a solution.

Over at the RO-Crate project Stian and Eoghan and I really liked the OCFL approach of having a clear spec with implementation notes, and we’re going to try to emulate that as we work on merging Research Object and DataCrate into one general purpose way of describing and packaging research data.

See also my presentation on DataCrate which introduces RO-Crate.

Trip Report - Open Repositories 2019 - Peter Sefton

2019-07-01T00:00:00+10:00

[Edited: 2019-07-01, 2019-07-2 fixed a few typos]

This year Open Repositories was in Hamburg, Germany. I was funded by the University of Technology Sydney to attend. I gave two presentations, one on our work on scalable research data repositories and other on research data packaging and ran a workshop, more on which is below.

This year was an intense, focussed conference for me. Last year I had a few take-aways from presentations; this year was one of those conferences where the value was in the conversations and in getting down to work.

Keynotes

I didn't get much out of the opening Keynote, from Jeff Gothelf, who's apparently a star in the User Experience (UX) world. It was a pretty generic presentation, with some well timed jokes but not very useful. Ok, UX is important, but when your examples are coming from well-resourced projects what are resource-constrained repository workers supposed to do to improve UX?

My mate Richard Jones from Cottage Labs asked a great question about how we can improve the UX for developers using APIs but didn't get much of an answer.

Even though I would have preferred a more on-topic keynote, it did spark a number of useful conversations in the breaks, including some I had with Richard about how we can build developer-friendly services. I always learn lots from Mr Jones at OR.

The closing Keynote was much more audience-appropriate. Heather Piwowar, who has, like, direct User Experience of Open Repositories gave us all a kind of career-affirming hug; pointing out that the repository people in the room are helping to build huge database of open scholarly information.

We put up and maintain the content, and her Unpaywall site finds it:

An open database of 23,565,946 free scholarly articles. We harvest Open Access content from over 50,000 publishers and repositories, and make it easy to find, track, and use. https://unpaywall.org/

She reckons our time is about to come - that we're at the take-off point for hockey-stick growth. Here's her presentation The Breakout Moment for open repositories is now. On topic, useful and inspiring.

Workshop in data packaging

Eoghan Ó Carragáin, Stian Soiland-Reyes and I ran an all-day workshop on Data Packaging which was well attended - we ran through the background in the morning, what data packaging is all about, why it's important and Eoghan gave us a tour of some of the major efforts going on in the space at the moment. Stian gave more detail on Research Objects, and Marina Soares E Silva talked about recent developments with Mendeley Data.

We structured the discussion around these themes or functions of Research Data Packaging:

Strategic: Transform scholarly practice, distribute the stuff that makes research complete, focus on redo-ability and reproducibility (eg Research Object) - Self-documenting: Describe the stuff in a programmer friendly way (eg FD, DataCrate), with a view to making it interoperable (eg DataCrate, RO), try to ensure Reusability (RO) - add human readable HTML (DataCrate, DataSpice)

Self-contained: Bundle stuff (eg Zip, TAR, .dmg, RAR).

Safe: Make sure the stuff is what it’s supposed to be using checksums (eg BagIt, FD) & Preservation / archival practice

Serialization Syntax: XML (Legacy systems), JSON (FD) Linked Data (RDF, JSON-LD eg RO, DataCrate)

Schemas - which one(s) to use?

We did a quick pass on these topics in the morning, looped back to the most popular ones in the afternoon - has some impromptu lightning talks and finished up with a session where we solicited Use Cases to go into the RO-Crate process Thanks go to Paul Walk and Neil Jefferies both of whom I've known for a decade or so for showing up and contributing enthusiastically.

My conference

I was lucky in my choice of converted water tower to stay in. I got to have breakfast with Martin Fenner from DataCite (we met back in 2011 at the first Beyond the PDF meeting). He was on the list for our Monday workshop but couldn't make it. I filled Martin in on what we're doing with RO-Crate - how we're using Schema.org metadata in JSON-LD. Martin was completely on-board with using Schema.org, and told me that if we have that, then using the DataCite XML format would probably be redundant. If that works out then that simplifies things - we need to talk more about this on the RO-Crate Project.

Martin also seemed to me to be frustrated that there are still a number of packaging efforts with no clear leader - simple, I said, just endorse RO-Crate when it comes out as DataCite approved :)

Stian and Eoghan and I took the opportunity to get stuck in to editing the RO-Crate draft, and getting to know each other a bit - forging a strong editorial team for RO-Crate.

I chaired a short-presentation session (7 minutes, 24 slides) - I like to get the presenters all down on stage at the end, get them to re-introduce themselves, give them a minute to tell us anything they forgot in the heat of presentation then get a discussion going.

Hamburg

Hamburg's a great place to wander around - lots of parks and lots of stickers on garbage bins that I really liked.

Hamburg harbour, where we went for the conference dinner boat ride, is a giant metaphor for Data Packaging. Container terminals as far as the eye can see being loaded and unloaded by giant robot-cranes. Shipping containers are, of course, a triumph of standardization and here you could see them being moved around at industrial scale - just like we're going to do using RO-Crate for Research Data.

Grant awarded; ARDC Discovery Activities; FAIR Simple Scalable Static Research Data Repository Demonstrator

2019-06-07T00:00:00+10:00

The eResearch team at UTS in collaboration with colleagues at QCIF and AARNet applied for and received funding under the Australian Research Data Commons "Institutional role in a data commons" grant scheme.

We applied for $50,000 but were only awarded $49,999 :(.

The proposal is reproduced below. We promised to share what we're doing - so there will be more coming about what this Oxford Common File Layout (OCFL)´ thing is, why it matters and what we're testing / demoing.

In short we want to explore how to store, preserve and publish large volumes of well-described research data using simple open technologies. We were already on this path and involved in international collaboration, the ARDC grant will help us get there quicker and test our ideas.

Title of the project: FAIR Simple Scalable Static Research Data Repository Demonstrator

Lead organisation/contractor: University of Technology Sydney

Project leader: Dr Peter Sefton – eResearch Support Manager

Project leader contact details: Email: peter.sefton@uts.edu.au

Phone: 0404 096932

Other organisations involved: AARNet – (Adam Bell) Queensland Cyber Infrastructure Foundation (QCIF) – (Andrew White)

Amount of funding requested (up to a maximum 50K): $50K

Proposal

Which area of fundable activity have you chosen:

Institutional data infrastructure, policies and procedures to support better research YES
Management of sensitive data YES
Integration of institutionally supported data infrastructure with national, discipline, and international infrastructure YES

Question(s) the project will address

In the interests of providing an improved procedure for resourcing, disposing and retaining data, can we demonstrate a FAIR research data repository architecture using static files laid out according to the Oxford Common File Layout (OCFL), an emerging international standards-effort that can operate sustainably at multiple scales from single-collection data sets to national collection? Can we also make this data Findable via a search portal using standards based metadata and make both open and sensitive data Accessible to the right parties, and improve tracking of data outputs managed inside and outside the organisation? Can OCFL increase opportunities for interoperability? Can distributing this data using the DataCrate specification increase the supply (and findability) of Interoperable and Reusable Research data objects with improved research data provenance? How might this architecture work with proprietary software such as Figshare and/or national services such as Cloudstor? How viable are these approaches for a range of disciplines? What other developments (e.g. standards for licensing sensitive data, procedures for applying access permissions at the file level for computing facilities, and national group management systems) would be needed to adapt this approach to storing and indexing sensitive data?

Proposed approach

We propose to build a demonstrator / proof of concept system which tests OCFL for use as a general-purpose data repository, for both open and sensitive data with a discovery portal. OCFL is chosen because it is based on established technology and can be used at computing facilities and on shared infrastructure without requiring server-based repository software or expensive and slow migration of large data collections via APIs.

The demonstrator will be populated with (1) specific datasets from the UTS data repository from a wide variety of disciplines including microbiology, history, computer science & speleology and of (2) varying scale from single collections to an entire university research data repository, building on the DataCrate for describing and packaging data, and (3) we will test the scalability of our approach by automatically generating a large number of plausibly-linked simulated test datasets and contextual entities (people, organizations, equipment, software describing data provenance) with group-based access permissions and demonstrate how a search portal can be used to ensure Findability and appropriate Access for the sensitive data by using an automated test suite to check the visibility of objects in a portal. We will also (4) demonstrate how individual data collections can be indexed in detail to produce collection-level discovery services, using two projects that were funded by the ANDS Major Open Data Collections: Farms to Freeways and Dharmae (UTS).

Who will be consulted/involved in the execution of the project and how will they be involved

This project will be led by Dr Peter Sefton at UTS who will also write the final report. eResearch Analyst Michael Lynch at UTS and staff at QCIF will develop specifications and software, building on preliminary work on a research data portal (for findability), extend it to ensure accessibility to the right users via a simple permissions system with static metadata that assigns group or individual access rights. The demonstrator repositories will be Interoperable with other software stacks using the same standard, particularly AARNet’s trial project “Adding Archival pathways to CloudStor” (investigating Archivematica as a preservation service) and via a project that is being proposed in Program 1 by the University of Melbourne. The project will consult with UTS stakeholders via the UTS eResearch Community of Practice which meets quarterly, via our regular eResearch outreach activities. We will consult with other Australian institutions via the ARDC network (we will offer to run webinars and keep the community up to date via mailing lists) and via our membership in Intersect. We will also consult with the international OCFL community.

Outputs/materials that will be shared with all of Australia as a result of the project

The project will result in open source code (both stand alone and as part of ReDBOX), open access documents; specifications and a report which answers the questions above in light of our findings. A project representative will deliver the report at the National Data Summit.

Evidence of ongoing commitment to outputs (if relevant)

UTS, along with QCIF, is one of the major contributors to the (originally) ANDS-funded ReDBOX Research Data Management Platform, and has demonstrated commitment to the ARDC community of REDBOX user institutions by contributing substantially to ReDBOX’s first major upgrade. UTS runs ReDBOX in order to support our strategic commitment to Research Excellence and Research Integrity and to implement our Research Management Policy - and will consult with the UTS Research Integrity officer Louise Wheeler. UTS is committed to building an OCFL based repository and discovery portal, but the work is not scheduled to be completed until 2020 - this funding will allow us to fast-track development of a demonstrator that can be presented to the ARDC, and the research community as a proof of concept for building a data commons at dataset, organisational, discipline and national scale. ReDBox is sustained by a community of Universities that pay a maintenance fee to the Queensland Cyber Infrastructure Foundation. The AARNet pilot project in this space will lead to sustainable investment should the demonstrator be successful.

Other information you wish to provide

Costing over 4 months is $50K (15 days of specification and design = $15K 30 days of coding = $30K 5 days of test-data development = $5K Report: (in kind) 5 days $5K) UTS is collaborating with the University of Melbourne on a proposal under Program 1, looking at the PARADISEC collection, and with AARNet investigating Archivematica & Cloudstor interoperating with OCFL & Datacrate packages, which complements this proposal. These projects together will aim to demonstrate the use of interoperable standards for building a data commons - testing a range of tools that operate over standardised (OCFL) repository architecture and standardised (DataCrate) metadata.

Hacky Hour - 1st Quarter 2019

2019-03-26T00:00:00+11:00

This Hacky Hour blog post covers the 14th February to the 21st March. These 6 sessions of Hacky Hour saw a good turnout, with both HDR students and researchers attending. Repeat visitors were common in the first few weeks – this was due to the solid assistance given by our eResearch team that encouraged attendees to return week after week, until they had solutions to their queries.

Queries that were addressed included:

HPC installation and use
Software expertise
GROMACs
SSH connection assistance
Redcap
Visualisation
STASH
Programming
Machine learning
Management of data supercomputer
Figures in R
Database assistance
GPU queries
Excel data processing
Geonomic / statistical analysis
Omero storage
General queries.

If you’re a researcher or higher degree research (HDR) student, then we warmly welcome you to join us at our Hacky Hour sessions every Thursday @ Penny Lane Café in Building 11 from 3-4pm. 🙂

History of Computational Computing

2019-03-12T00:00:00+11:00

Last week I attended the Dr Trevor Pearcey Centenary Celebration. This was an afternoon of talks at Sydney University to celebrate the achievements of Dr Trevor Pearcey, a British-born Australian scientist, who created CSIRAC in 1949. This was Australia's first digital computer and the 4th or 5th stored program computer in the world. CSIRAC is the oldest surviving first-generation electronic computer in the world. I saw CSIRAC a few years ago when in Melbourne.

In comparison the talk by Dr Sarah Pearce, Deputy Director of CSIRO Astronomy & Space Science, covered the computer at the Pawsey Centre to process the data from the Australian Square Kilometre Array Pathfinder Telescope. The ASKAP is already producing 5.2 terabytes of data per second, which is processed on site and then streamed down to Galaxy and stored at the rate of 2.5 Gbytes per second.

	CSIRAC	Pawsey's "Galaxy"
Number of cores:	1	9440
Clock Speed:	1 kHZ	3.00 GHz
Operations/second:	1000	200 million million
Memory:	~ 1K or 2K	31.55 Terabytes
Local storage:	1024, 20 bit-words	1.3 Petabytes
Logic elements:	2000 valves	7000 billion transistors
Input:	punched paper tape	high level computer language

Prof. Hugh Durrant-Whyte, the New South Wales Chief Scientist & Engineer, also gave a talk on "Why Science Matters", Barbara Ainsworth, Pearcey Biographer & Curator of the Monash Museum of Computing History spoke about Pearcey's history and his innovations.

Prof. Andrew Dzurak (Director of the Australian National Fabrication Facility & ARC Centre of Excellence for Quantum Computation & Communication Technology, UNSW) and Prof. David Reilly (Director of Microsoft Quantum, Sydney & Chief Investigator in the ARC Centre of Excellence for Engineered Quantum Systems, School of Physics, University of Sydney) both explained to us why quantum computers will be important and the current state of development of this technology. Dr Baerbel Koribalski (OCE Science Leader, CSIRO) gave us a great talk on how super computers have enabled astronomers to model galaxy formation across the life and physical span of our universe.

Prof. Ben Eggleton, Director of the University of Sydney Nano Institute, School of Physics, University of Sydney also gave some participants a tour of the Nano Labs.

References: CSIRAC, The First Computer in Australia, 1949-1964; Pawsey Supercomputing Centre's Galaxy; Australian Square Kilometre Array Pathfinder Telescope

Mike Lake
eResearch

PBSWeb Released as Open Source

2019-01-23T00:00:00+11:00

I have released as open source the small web application that we use to show the nodes, queues and jobs running on our HPC cluster. This is used by the cluster administrators to quickly see the status of the nodes and how busy they are, to check how full each queue is and to see what jobs are running and queued. It’s also useful for users to see this information as well.

You can see this web app in use here: https://hpc.research.uts.edu.au/status/
It's been released on GitHub here: https://github.com/UTS-eResearch/pbsweb
And it's been announced as available on the PBS Pro open source forum.

Mike Lake
eResearch

Upcoming Training Courses in 2019 Have been Updated

2019-01-22T00:00:00+11:00

Upcoming Training Courses in 2019 Have been Updated

UTS eResearch and Intersect offer a wide range of specialised courses for researchers, from beginner through to advanced levels in High-Performance Computing (HPC), Programming with R/Python/Matlab, Excel, data management, data cleaning and visualisation, databases and SQL, and more. Delivered by Intersect's team of experts, training courses provide practical and research-relevant hands-on exercises.

Upcoming training courses are updated regularly. The latest update has been done on 22 Jan 2019. Courses from March are not open for registration yet, please keep an eye on our training page.

Provisioner - A Framework for Integrated Research Data Management

2018-12-03T00:00:00+11:00

This is a presentation by Mike Lynch, Peter Sefton and Sharyn Wise, delivered at eResearch Australasia 2018 by Mike Lynch.

Mr Michael Lynch Dr Peter Sefton Ms Sharyn Wise ' title='A Framework for Integrated Research Data Management With services for planning, provisioning research storage and applications and describing and packaging research data

Mr Michael LynchDr Peter SeftonMs Sharyn Wise ' border='1' width='85%'/>

A Framework for Integrated Research Data Management ' title='Provisioner Integrate research data management into research applications Allow researchers to self-provision research apps Apply lessons learned from earlier generations (Data Capture) Give researchers something, get out of the way We didn’t want to build a monolith Small parts, loosely joined, data-centric

A Framework for Integrated Research Data Management ' border='1' width='85%'/>

Notes - Slide 2

Provisioner grew out of two conflicting requirements:

We want to be able integrate research data management into the tools researchers actually use to do their research, giving the researchers something besides data management – access to facilities and software, easier publication, etc
We didn’t want to build a monolith, and wouldn't be allowed to if we did

Notes - Slide 3

A real case of complex data management.

Our original idea was to put a data repository in for Data Arena customers, from which the DA team would download datasets.

But the DA team just wanted NFS mounts, and repositories aren’t a good fit for large datasets - high-end visualization and data science both use filesystems.

What they could use: a git repository to manage their code and pipelines (GitLab)

On the other end, the MIF have long wanted to use OMERO for microscopy.

Our original plan was for a point-to-point integration which would only service two flagship installations.

The key insight out of which Provisioner grew was: what if we could connect every research app to every other one? (eventually)

A Framework for Integrated Research Data Management ' title='Apps and workspaces App = a research application – can be specific or general OMERO – dedicated microscopy app GitLab – for research software development Coming next: file shares, ELNs Workspace = a research team’s thing within an app A workspace is linked to an RDMP and an owner Once it’s created, the researcher goes directly to the app

A Framework for Integrated Research Data Management ' border='1' width='85%'/>

Notes - Slide 4

Note that the term "workspace" is already being used by another UTS project – we were gazumped – but we still think it’s the best terminology for the abstraction we're trying to describe.

Create a new workspace for an RDMP Share a workspace with colleagues Export data (to a data record or another workspace) Import data

We are building this out incrementally: create and export are first

A Framework for Integrated Research Data Management ' title='Workspace API The workspace abstraction lets us capture common high-level operations:

Create a new workspace for an RDMP Share a workspace with colleagues Export data (to a data record or another workspace) Import data

We are building this out incrementally: create and export are first

A Framework for Integrated Research Data Management ' border='1' width='85%'/>

A Framework for Integrated Research Data Management ' title='ReDBox 2.0 RDMP and dataset description tool Improved metadata collection and integration Service catalogue for provisioning workspaces in apps Data publication workflow Modern re-implementation (Node.js, Angular, Mongo) More maintainable No more curation – web rather than db principles

A Framework for Integrated Research Data Management ' border='1' width='85%'/>

Notes - Slide 7

Metadata integration takes a "just-in-time" approach, where it's collected only when it's needed.

Each metadata record is derived from the previous one:

Research data management plan
Data record
Data publication

Notes - Slide 8

On the left is the original design, which had a separate Provisioner component (in blue) which orchestrated api requests and talked to the research apps

On the right is the product we built, with no separate orchestrator, and modules within ReDBox which drive the apps

What happened to that orchestration layer?
Needed to break the abstraction to authenticate to GitLab
This is what happens when concepts for APIs hit real life
The minimum viable product didn’t end up including it
This is a better design: simpler, and we can implement orchestration/queueing if and when it’s needed

Notes - Slide 9

At present, we have public datasets in our ReDBox 1.9 instance being fed to RDA.

Data publication is manual: once the metadata is ready we (actually, I) put it on an nginx web server

The new model isn’t that different from this: the repository is a filesystem with static DataCrates. The HTML catalogs in these act as landing pages.

We build a solr index from the machine-readable metadata in the DataCrates, and the user interface to this is a single-page Angular app.

Much better for security: we can host the website on a public-facing VM with just nginx and solr and keep ReDBox inside the firewall

Immutable datasets – we’re in discussion with a research group who want to publish their live data – large networking datasets which they’re in the process of refining.

Try to allow them to do this transparently but still allow integrity, so a given DOI points to a determinate snapshot of their data.

Oxford Common File Layout for the static DataCrate repository

Notes - Slide 11

This is a diagram drawn by Gerrad Barthelot, head of our architecture team.

It shows where we’d like to be in a few years – with an expanded range of services and research apps with Provisioner adaptors to Stash 3 (ReDBox 2).

Note that this diagram shows the original Provisioner design with an orchestration layer between ReDBox and each research app.

For now, we’re continuing with the simple model of ReDBox hooks acting as adapters directly talking to apps, and we’ll use middleware between these and the apps if and when we need it, on an app-by-app basis.

Not all of the connections will be fully automated – some may put a request into ServiceConnect, or even just send an email

' title='Thank you

' border='1' width='85%'/>

Launching DataCrate v1.0 a general purpose data packaging format for research data distribution and web-display

2018-11-01T00:00:00+11:00

This is a presentation by Peter Sefton, Michael Lynch, Liz Stokes and Gerard Devine, delivered at eResearch Australasia 2018 by Peter Sefton.

' title='Launching DataCrate v1.0: a general purpose data packaging format for research data distribution and web-display

' border='1' width='85%'/>

Notes - Slide 1

In this presentation we will launch version 1.0 of the DataCrate standard. The presentation will cover:

The motivation for this work, and prior art - why we needed to bring together the standards we did in the way that we did.
A walk-through of example data crates from a variety of sources, speleology, clinical trials, simulation, social history, environmental science and microbiology.
An introduction to tools for making data crates with an appeal to attendees to join us in making more tools, for more new kinds of data.
A demonstration of how DataCrates are being used at UTS to move data though the research lifecycle - archiving and publishing data.

Notes - Slide 2

The following people contributed to this presentation:

peter.sefton@uts.edu.au
michael.lynch@uts.edu.au
elizabeth.stokes@uts.edu.au
g.devine@westernsydney.edu.au

Notes - Slide 3

There were no existing generic data packaging standards with human and and machine readable

🙅 === FACE WITH NO GOOD GESTURE

Notes - Slide 4

Our motivation was to be able to display and distribute data sets with useful "who, what where" metadata in a way that is easy to for coders to target, and easy for researchers to consume, both as readers and programmers who might want to run code against a data set.

Notes - Slide 5

We have a growing list of examples.

Notes - Slide 6

DataCrate provides human-readable HTML data about files including detailed metadata.

' title='Ability to describe file provenance

' border='1' width='85%'/>

Notes - Slide 7

This slide shows a CreateAction, where an instrument - a Lidar scanner - was used by an agent - the person - to create two files.

' title='Software can be an instrument too

' border='1' width='85%'/>

Notes - Slide 8

This shows a software package (instrument) acting on a file (object) used to create another file (result) - a sepia version of a picture.

' title='All metadata is available in JSON-LD

' border='1' width='85%'/>

Notes - Slide 9

DataCrates contain metadata in JSON-LD.

' title='... so relationships can be visualized

' border='1' width='85%'/>

Notes - Slide 10

Why do we want machine readable data? One reason would be to generate visualisations that help people understand relationships in the data set. Here’s a demo I coded up in about half-an hour before the conference that shows how we might visualise the the way files are created. It shows a Person (me) who is the agent in two CreateActions, one where the instrument is a camera/lens combination and the object is the place being pictured, and the result is a file, and one where the object is said file, the instrument is a software package, and the result is a sepia version of the original photo.

' title='URIs as names for things

' border='1' width='85%'/>

Notes - Slide 11

Each term used has a link to its definition, eg: https://schema.org/CreateAction

Notes - Slide 12

Tooling is a problem. JSON-LD is a great format, but: There are no utility libraries for things like looking up context keys.

Notes - Slide 13

Calcyte uses multi-worksheet spreadsheets for data entry, based on an idea of Mike Lake’s.

' title='

' border='1' width='85%'/>

Notes - Slide 14

This works, but it’s not an ideal user iterface.

Notes - Slide 15

Gerard Devine is developing a tool which will allow DataCrate export from the Australian National Data Service funded HIEv system.

HIEv DataCrate - At the Hawkesbury Institute for the Environment at Western Sydney University, a bespoke data capture application (HIEv) harvests a wide range of environmental data (and associated file level metadata) from both automated sensor networks and analysed datasets generated by researchers. Leveraging built-in APIs within the HIEv a new packaging function has been developed, allowing for selected datasets to be identified and packaged in the DataCrate standard, complete with metadata automatically exported from the HIEv metadata holdings into the JSON-LD format. Going forward this will allow datasets within HIEv to be published regularly and in an automated fashion, in a format that will increase their potential for reuse.

' title='

' border='1' width='85%'/>

Notes - Slide 16

Christian Evenhuis is developing a tool for exporting microscope images from Omero.

' title='

' border='1' width='85%'/>

Notes - Slide 17

Chris is working to describe the equipment used in the Microbial Imaging Facility (MIF), Here’s a page for a microscope, this is part of work in progress to descibe as much of the context of research in MIF as possible.

' title='

' border='1' width='85%'/>

Notes - Slide 18

Peter Sefton has developed code to export Omeka Classic repositories to DataCrate. This is an example of one from the University of Western Sydney curated by Katrina Trewin. This uses the Portland Common Data Model for modelling repository structure. We are using these data sets to help develop an Omeka service based on the Omeka S software, along with data from Dspace extracted using another nascent code project.

' title='

' border='1' width='85%'/>

Notes - Slide 19

Provisioner grew out of two basic requirements, which seem to conflict with one another:

We want to be able integrate research data management into the tools researchers actually use to do their research, rather than as an add-on to an existing process (like DC)
Any such system should give the researchers something besides data management – access to facilities and software, easier publication, etc
We don’t want to build a monolith, and even if we wanted to build a monolith, we wouldn’t be allowed to – the current mood is SAAS, on-premises only if necessary, no single points of failure
The UNIX philosophy of small parts, loosely joined, and the idea that data has gravity

' title='It’s standards all the way down Oxford Common File Layout ← Static file-based repositories THIS TALK → DataCrate ← THIS TALK Data Crate builds on Bagit ← Data packages w/ checksums, content by ref Schema.org ← Main metadata standard / Repo metadata standard → PCDM JSON-LD ← Linked data in programmer-friendly format

' border='1' width='85%'/>

Notes - Slide 20

' title='

' border='1' width='85%'/>

Notes - Slide 21

Next step is to take this to an international meeting to see if we can get some agreement between project using similar approaches.

' title='

' border='1' width='85%'/>

Notes - Slide 22

Dataspice does a similar thing to DataCrate - they could easily be aligned.

' title='

' border='1' width='85%'/>

Notes - Slide 23

Research Object Bundle also tries to package data with JSON-LD data, but in a way that is (we think) more complicated to implement, and without the human-readable web-site embedded in the package.

Critique the standard

Generate some more sample data sets as a spec for people who will ...

... write a packaging tool
Export from data management system (eg MyTardis :)
Write a GUI or web tool for people to create DataCrates
Help add viz to our HTML pages

' title='Help wanted! We invite you to: Critique the standard Generate some more sample data sets as a spec for people who will ... .. write a packaging tool Export from data management system (eg MyTardis :) Write a GUI or web tool for people to create DataCrates Help add viz to our HTML pages

' border='1' width='85%'/>

Notes - Slide 24

Please help.

' title='

' border='1' width='85%'/>

Notes - Slide 25

Please contribute to or use the spec

LinuxConf 2018

2018-09-30T00:00:00+10:00

The 2018 Linux Conference Australia was held the University of Technology Sydney from 22-26 January 2018. I attended courtesy of eResearch.

Linux totally dominates supercomputers. As of November 2017, all 500 of the world's fastest supercomputers were running Linux. This is because most of the world's scientific software for generating or crunching research data is written to run on Linux systems because of its openness, ability to be customised, speed and robustness. The eResearch HPC runs the Centos 6.9 Linux distribution and some of our users run a linux "distro" themselves.

Like most large conferences there were many parallel sessions, so deciding what to attend was sometimes a difficult choice. The sessions I attended mostly reflected my interest in bioinformatics and system administration, as those topics have relevance to the HPC cluster administration and user support.

Of particular interest in the bioinformatics stream was; James Ferguson and Dr Martin Smith of the Garvan Institute and their nanopore sequencing pipeline, the data and code that Stemformatics and CSIRO's bioinformatics provide, Matt Tods keynote on the Open Source Malaria project and key requirements of lab notebooks (read some of their open Lab Notebooks).

In the system administration stream of particular interest was Terraform for building and versioning infrastructure safely, James Shubin's new configuration management tool Mgmt, using mod-security and fail2ban, and Thomas Schöbel-Theuer's MARS Long Distance Replication which replicates huge amounts of data across continents.

The talk on high performance science covered data formats, provenance and reproducible research. If your using git and don't mind a good command line tool for reproducible research then have a look at DataLad. This is build on top of git-annex and extends it with an intuitive command-line interface. ReproZip allows you to pack your research along with all necessary data files, libraries, environment variables and options. It's a python package similar to recipy which also attempts to help with the problem of reproducible research.

Mentioned was an interesting site binder, which turns your GitHub repo into a collection of interactive notebooks (consider security though). If you use pipelines for your data flow then here is a curated list of awesome pipeline toolkits, apparently inspired by a curated list of amazingly awesome open source sysadmin resources.

Other personally interesting talks were the MUON detector, and stories from The Register's "On Call" and, if you are an open source developer, an excellent talk on the problem of contributor overload.

LinuxConf 2018 Stats: 200 presentations selected from 400 submissions, 700 attendees from 20 countries, 140 MB/sec sustained bandwidth for 1.8 TB during the conference, 1321 unique network devices detected, and more than 2000 coffees served.

The next Linux Conf will be held from 21-25 January 2019 at the University of Canterbury, Christchurch, New Zealand. See the link here: LCA 2019, Canterbury NZ. Registration costs for students and hobbyists is always kept low and Linux Confs are always very inclusive as diverse groups of people are made welcome.

Mike Lake
eResearch

Domesday Preppers! Prepping for preservation at University of Technology Sydney eResearch, ITD

2018-08-02T00:00:00+10:00

This presentation was written and delivered by Sharyn Wise (with couple of slides from Peter Sefton) for the Australasia Preserves meeting at the NSW State Library. This was one of a number of short talks from various organisations and was the only one to focus on research data. UTS has not started our Digital Preservation journey yet - we're still packing our things and looking at the map, or 'prepping'.

Notes - Slide 1

Today I’ll be presenting on preparing for preservation at UTS, because we are not there yet, and I suspect it is more of a journey than a destination anyway. Preservation is on our eResearch Roadmap for next year, so I am grateful to learn from the experience of others today. I named this talk in a rather feisty moment, thinking that the story of the Domesday book was the archetypal cautionary tale and would be well-known in preservation circles. If you don't know the story you can google it, but the title in the Guardian says it all – digital data is incredibly fragile in comparison to earlier technologies. Instead I will begin with what I think of as a Domesday tale of research data.

' title='Why preserve? A research data ”Domesday” tale

' border='1' width='85%'/>

Notes - Slide 2

I still remember seeing media headlines about the potential dangers of unbalanced Omega 6 oil consumption in 2013 – like this one here in Time Magazine. But as so often happens with media headlines, the back story of the actual research paper in BMJ turned out to more complex and interesting. A U.S Cardiovascular Health researcher, conducting a meta-analysis had come across a paper from a 1973 clinical trial, the Sydney Diet Heart Study which according to the lipids hypothesis of the time, had replaced saturated fat with polyunsaturated fat in the diet. This trial had been discontinued because the participants had shown a higher rate of mortality. However, when the Sydney researchers wrote up the results, they had not reported on specific causes of death. The data confusingly showed the opposite of what they had hypothesised, so they published a brief paper and left it at that.

It just so happened that the unsaturated fat used was safflower oil - which we now know uniquely contains only Omega-6 fatty acids. So the US researcher began hunting for the dataset. He eventually tracked down the last surviving member of the research team and asked if the data was available. Luckily the Sydney researcher had kept it among piles of boxes in his garage. After a bit of rummaging, he produced an obsolete 9-track magnetic computer tape. Data Recovery was expensive and difficult, but worth it, because of what it added to knowledge of lipid metabolism: the Omega 6 group had 6 %age points higher risk of death from cardiovascular and coronary heart disease than the control.

Now let's remember that this study could not be replicated without risk to human life so that data would have been gone forever if not recovered. I like this cautionary tale, because it highlights some extra challenges we face. Like how do we know what to preserve, when knowledge grows in ways that cannot always be anticipated? Of course there is the Records Act which requires that research data from human clinical trials is preserved for up to 25 years, but that wouldn’t have been long enough to save this data. Our Records Managers suggest that it should be up to the researchers, as the experts, to recommend their data for longer term preservation. However, would the Sydney Heart Diet trial researchers have selected this data for preservation? Probably not. So do we preserve everything? Well that isn't really financially viable – as we know, its one thing to keep data and quite another thing to manage it over years. And that brings me to the next challenge.

Notes - Slide 3

Which is getting research data under management in the first place. Researchers are suspicious of giving their data to anyone – after all it is their competitive advantage. So lets step back - What is research data?

It can be just about anything – which in itself is daunting. A lot of research data is not human readable, like the last two of these three examples – geospatial data and microscopy data. So obviously we need to preserve more than just the outputs if the data is to remain accessible and usable. Broadly speaking, the usual approach here is to convert to open formats, ideally without losing metadata, and to preference open source software if possible, which we can retain. And this is where the concept of preparing for preservation comes in.

We need to start a lot earlier in the data lifecycle than the archiving stage, to ensure that metadata and provenance data are not lost. And because this may involve changes to researcher practices it is also a huge cultural change challenge. One problem we face is that most scientific instrument data comes off instruments in proprietary formats. This problem is being addressed in Optical microscopy by a consortium called OME - Open Microscopy Environments, and Bioformats, who work to bust open proprietary files from optical microscopes, and extract the metadata as well as the images as TIFs. The microscopy slide shown here is inside Omero, their open source imaging repository. Similarly, the map data is in open standardised formats inside Geoserver, an open source geospatial platform. In both cases, these platforms offer significant enough benefits to researchers that they are willing to use them as repositories for their data. So does Gitlab, the code repository environment shown here. So far so good.

Provisioner

Provisioner Research Data Mgmt Plan Research Data Catalogue

Picture credit: Gerrad Barthelot, Technical Architect, IT Infrastructure UTS ' title='6

Provisioner

Provisioner Research Data Mgmt Plan Research Data Catalogue

Picture credit: Gerrad Barthelot, Technical Architect, IT Infrastructure UTS ' border='1' width='85%'/>

Notes - Slide 4

But how do we bring together these discipline specific repositories into one managed solution? By loosely coupling them together in an architecture we are developing called the Provisioner. A researcher (top centre) accesses the data management catalogue “Stash”, which provisions managed workspaces at the beginning of their projects and at the end, creates data records, where they can upload or link to their data in place. Stash keeps all the various metadata packages we need to manage, describe, contextualise and ultimately preserve the data and as we find suitable research platforms to add, we will write an adaptor to add it in.

Images sourced from Wikimedia Commons ' title=' Where is the data?

Persuade researchers that 'keeping' is not 'managing'
make archiving easier with DataCrate.

Images sourced from Wikimedia Commons ' border='1' width='85%'/>

Notes - Slide 5

So what about smaller, human-readable datasets – like surveys, or heterogeneous humanities data collections, for example? It can be hard to persuade these researchers of the value of managing their data too, rather than throwing it on some media and into a drawer. Even worse is that even for those who want to manage their data properly, there has been a dearth of simple tools for researchers to capture the necessary metadata and bundle their data for archiving, as Cameron Neylon complains here on his blog.

Notes - Slide 6

Enter DataCrate, which is a new standardization effort lead by UTS, specifically for research data. The specification is available on GitHub, we’d welcome your input: feel free to raise an issue with suggestions or disagreements, or to send us a pull request with changes.

The spec is designed for implementers, it explains how DataCrate is built on existing standards, the BagIt packaging specification, and how to add Linked Data metadata in JSON-LD format to describe the package, its files, and its context, such as the people who created it, the organisations which funded it, the equipment used in creating the data and so on.

The plan is that DataCrate will allow us to move data between systems (including by-reference using BagIt’s link features) with rich file-level and package level metadata, with human-readable manifest travelling with the data to help contextualize it for future users.

Notes - Slide 7

This is a screenshot of a data set available here

If sufficient of this metadata is present, then the spec explains how to construct a DataCite citation as we can see here.

Essentially, DataCrates are data packages that have a content-manifest with checksums, to help ensure data-integrity, they have metadata in a linked-data format (using JSON-LD and metadata terms from schema.org and other ontologies) and they have an index.html file which displays the metadata in a human readable summary format, both for the package (crate) and …

Notes - Slide 8

... they have File-level descriptions of the content. So although the original use case I just mentioned was packaging data for retention, we are prepping for preservation here also by employing Richard Lehane's Ziegfried tool to run through several of the steps required to identify preservation metadata.

(The screenshots here are from a data set containing cave survey data and the UK national Archives’ Pronom format registry)

Acknowledgements STASH and Provisioner development is supported by the Australian National Data Service (ANDS) through the Australian Government’s National Collaborative Research Infrastructure Strategy Program and the Education Investment Fund (EIF) Super Science Initiative. STASH is based on ReDBox, developed by Queensland Cyber Infrastructure Foundation Ltd (QCIF) Sharyn Wise – eResearch Analyst sharyn.wise@uts.edu.au Peter Sefton – Manager, eResearch Support peter.sefton@uts.edu.au Data Crate - https://github.com/UTS-eResearch/datacrate/

Authors ' title='

Authors ' border='1' width='85%'/>

Notes - Slide 9

So in sum, at UTS we may be early on the journey to preservation, but I've tried to illustrate some ways that we are building preparation into our systems and thinking so that when we come to implement preservation workflows, the means to do so, metadata and tooling for example, will be there. Here are our contact details and we will be happy to answer questions at any time. Thank you

Research Bazaar Sydney 2018 Report

2018-07-24T00:00:00+10:00

A couple of weeks ago, I attended Research Bazaar (ResBaz) Sydney 2018 as an instructor. This is a wrap-up report on this successful and enjoyable event.

Overview

Research Bazaar was initiated in Melbourne, and now it is a worldwide festival promoting the digital literacy emerging at the centre of modern research, where researchers can learn from training courses, share knowledge and skills, network with peer researchers and have fun. In Australia, ResBaz has been held in Melbourne, Sydney, Brisbane, Perth and Hobart. Sydney’s first ResBaz was held at the University of Sydney in February 2016, which was aimed mainly at researchers from the University of Sydney. Luckily, at that time I was invited to be one of the trainers, which allows me to be a witness of the growth of this event in Sydney. Last year, with the support of our UTS eResearch team and the UTS library, ResBaz was held at UTS in July 2017. This was essentially the first “ResBaz” ResBaz with the involvement of multiple institutions: University of Technology Sydney, University of New South Wales, University of Sydney, Macquarie University, and Intersect. With 300+ registrations and 13 workshops delivered, the event was quite a success. I acted as an instructor again.

Earlier this year, five Intersect colleagues, and myself became Software Carpentry instructors. Now Intersect has eight Software Carpentry accredited trainers, plus some experienced eResearch analysts and trainers to support ResBaz moving forward. ResBaz Sydney 2018 was held at Macquarie University on 3-5 July 2018. This community-building event keeps growing over time - over 600 researchers and research technologists from 9 universities (Macquarie University, University of Sydney, University of New South Wales, University of Technology Sydney, Western Sydney University, Australian Catholic University, University of New England, University of Newcastle, University of Wollongong) registered for this year’s sessions (compared with 300 registrations from 4 universities last year), including 78 registered researchers from UTS.

Training & Talks

Training was the key part of ResBaz that attendees enjoyed so much. Together with Aidan Wilson (Intersect) and Robert Woodward (Macquarie University), I co-instructed in one of the two Python streams over two days that covered Unix, Python and Git. There were other sessions including Beginner R Programming, Advanced R and Python Programming, Humanities and Social Sciences, SQL & Data Manipulation (including web scraping). The “themed” lightning talks included the Humanities and Technology Camp Unconference, Machine Learning, Lean Innovation for Researchers, etc., and there were sponsor talks on the third day.

Me Teaching Python using Jupyter Notebook :)

Researchers Coding!

As a witness and instructor in all the 3 ResBaz Sydney events in history, I found the following highlights of this year’s training:

Macquarie University used an “overbooking” strategy for all training streams. It worked quite well to get a better turnout. In our Python stream, 33 researchers turned out, which was better than last year.
More experienced instructors and helpers than last year shared the teaching load in each session. There were three Software Carpentry accredited instructors and three helpers in our Python session with 33 attendees, so the trainer-trainee ratio was quite high.
We’ve achieved absolutely positive feedback. The only “negative” feedback was that the researchers wanted to have shorter breaks and more time for learning!
Some lightning talks were quite inspiring and were well-received.

Bazaar Spirit

The grand hall was used for networking, session break catering, posters and sponsor booths. Although catering was not stellar, we all had a great time talking with researchers about their research and posters, where and how they can get support, and reunited with old friends and colleagues.

One highlight was that Intersect provided popcorns and Intersect@10 anniversary cupcakes! They were so popular and attendees loved them.

Coffee break

Popcorn Served by Intersect

Learning Git with Intersect@10 Cupcakes

Reunion - Sharyn (UTS), Piy (Usyd) and Carmi (CSIRO)

Reunion - the Intersect Team

One thing that could be improved in the future is that the main hall was not close to one of the two training buildings. It was a bit hard for attendees to move back and forth to the catering/booth area - this could be one of the reasons why they didn’t like the session breaks. Best to have all training rooms in the same building, so participants can get out of the rooms during breaks and grab a coffee and muffin, and get back to rooms easily whenever they want.

Support On!

ResBaz is about eResearch, research community building, and research technology support. This year, the participation of regional universities have highlighted the successful event - this is a sign of ResBaz expansion and spirit spreading!

Not only did I represent Intersect, I was also representing UTS as part of the UTS eResearch team. I was so delighted to see many UTS researchers attend this event, and benefit from the training and networking. This is in addition to our training courses provided at UTS and the Hacky Hour eResearch support by the UTS eResearch team that UTS researchers already have access to.

Last but not least, Intersect was one of the biggest supporters of ResBaz Sydney 2018. On top of being a major sponsor, one Intersect eResearch Analyst sat in the organising committee; 11 people were sent to ResBaz as instructors and helpers (some of them came to Sydney from Melbourne, Geelong, Canberra, Armidale and Newcastle), and six training workshops were led or co-instructed by Intersect.

I am looking forward to the next ResBaz Sydney in 2019! Which university will volunteer to host it? Stay tuned!

Need Training and Support @UTS?

Training courses are updated regularly. Please keep an eye on our training page. The latest update has been done on 19 July 2018.

Weekly Hacky Hour tech support: 3-4pm every Thursday @Penny Lane (Building 11).

Trip Report (with bonus opinions) - Open Repositories 2018, Bozeman Montana, USA

2018-07-10T00:00:00+10:00

I (Peter Sefton) recently attended OR2018, the Open Repositories conference from June 4-7, 2018 in Bozeman Montana.

This post is being posted on the UTS eResearch site and on my site.

My trip was funded by the University of Technology Sydney (UTS).

Mission

Gavin Kennedey from QCIF was also in attendance, and we were on something of a mission - to promote and get feedback on the recent work we've been doing on the ReDBox research data management platform. We ran a ReDBOX intro workshop on the Monday of the conference. Gavin and I also presented a general introduction to ReDBOX and the provisioner, and I went into more detail about the DataCrate standard for shipping and showing-off research data that I have been leading, with help from a growing community of supporters. I also did a presentation in the technical session which included a live demo of using ReDBox to ingest DataCrates - showing how it could 'sniff out' metadata from DataCrate packages.

Research Data Management

Open Repositories now has enough going on about Research Data Management that I was able to spend most in those sessions when I wasn't meeting with people directly.

I heard people talk about a few things that helped confirm some of our design choices at UTS, and a few things to challenge my world view as well.

Vladimir Bubalo from Macquarie, Gavin and I chased up some more details about the Dataverse repository software.There is some interest from Macquarie in what's available in the way of open source research data repositories.

As attested by the session Reaching out with Data: Dataverse Creating a Global Community Dataverse looks like it's a thriving product and would be a good integration target for ReDBOX, it powers the Australian Data Archive for one thing and can be used as an institutional data repository. One problem they're still grappling with is large file support, which is the same issue with any repository software when you try to put large volumes of, or large numbers of data streams through the API or web interface.

There was a really interesting talk from the UK Jisc Research Data Shared Services (RDSS) project (on which I have done some very part time consulting via Artefactual in Canada) about how they failed to get a Samvera repository working as part of their offering.

The have published their report.

It seems like the project to adapt Samvera failed not because of large files or large volumes of files, although that problem may have come up later, as we reported at OR2014, Hydra, which was rebranded Samvera had severe performance problems on the Alveo virtual lab related to processing number of files in a single transaction. Two issues Jisc called out in their report are one, a failure to implement their complex domain model in Samvera. They say:

In order to capture and store the range of metadata required by the RDSS CDM the internal storage model of the work type within RDSS Samvera needed to map closely to the CDM’s conceptual data model. However, it did not prove to be straightforward to translate the conceptual data model of the RDSS CDM, which leverages entity-relationships, to a programming/storage model that is largely intended to be flat.

And two, they had problems implementing a message passing model.

In order to avoid the kind of problems reported by Jisc and to deal with the reality that many researchers just want to gulp-down files, or object-store blobs, or database tables and most emphatically do not want to sip data through a tiny API-straw we are building research data management systems that are distributed:

Avoiding complex customisation of applications wherever possible and instead linking Research Data Management Plans (RDMPs) to workspaces in research software such as lab notebooks or git repositories, and archival data stores using standard linked data techniques.
Not using message-queues that might block progress on workloads (lesson learned from ReDBOX 1 development and its ill-conceived "curation" workflows). We plan to run "bots", stand alone agents instead which generate reports or make links or auto-provision spaces.
Keeping archival data, and data that needs rapid access in large scalable file storage systems, described and linked to RDMPs, data descriptions, and data publication records. For us, 'the repository' is a collection of services which includes the central index and a variety of stores, rather than a single application: It's a lifestyle; it's not just for Christmas; and most importantly it's a governance thing.

Given the above one of the things I was keen to do at OR2018 was to find out more about the Oxford Common Filesystem Layout (OCLF) which is being driven by people from Oxford, Emory, Stanford, Cornell and Duraspace - it's about how you organise digital assets in the kind of deconstructed architecture I described above for UTS. The files are kept on a file system (yes cloud-first people, that might actually be backed by an object-store) so that you can run services against them: index them for discovery, check their integrity, generate dissemination versions for distribution, report on items due for disposal and so on.

I couldn't get to the OCFL presentation but lead author Neil Jeffries from Oxford talked me through the emerging standard and the process being used to develop it. Neil assured me we can go ahead and implement against the draft spec (and challenged me with the statement that at Oxford they don't like to talk about data vs metadata, it's all data. I'm still thinking that over Neil, but I think I still believe in metadata).

This is from the (still rather sparse) draft dated 2018-06-22, a couple of weeks after the conference:

A general observation is that the contents of a digital repository -- that is, the digital files and metadata that an institution might wish to manage -- are largely stable. Once content has been accessioned, it is unlikely to change significantly over its lifetime. This is in contrast to the software applications that manage these contents, which are ephemeral, requiring constant updating and replacement. Thus, transitions between application-specific methods of file management to support software upgrades and replacement cycles can be seen as unnecessary and risky change, changing the long-term stable objects to support the short-term, ephemeral software.

By providing a specification for the file-and-folder layout on disk, the OCFL is an attempt at reducing, or even eliminating, the need for these transitions. As an application-independent specification, conforming applications will natively 'understand' the underlying file structure without needing to first transition these contents to their own format.

https://ocfl.io/

From my notes of the conversation with Neil:

Multiple things accessing the file system writing changes to LOGS. Eg fixity check, or create new.

There is no state. File system is the state. Digital pres services are workers / crawler or message queue driven.

The incomplete spec is online - it's not enough to do a complete implementation yet, but we will track it.

The host university, Montana State had a nice take on this too. Sara Mannheimer, Jason A. Clark, James Espeland presented A Prototype for the Institutional Research Data Index

Most out-of-the-box institutional repository systems don’t provide the workflows and metadata features required for research data. Consequently, many libraries now support two institutional repository systems—one for publications, and one for research data—even when there are nearly a thousand data repositories in the United States, many of which provide services and policies that ensure their trustworthiness and suitability for institutional research data. Libraries are either increasing spending by purchasing data repository solutions from vendors, or replicating work by building, customizing, and managing individual instances of data repository software. This presentation suggests a potential solution to this issue: a prototype for an open source Institutional Research Data Index (IRDI) that promotes discovery and reuse of institutional datasets through automatic metadata harvesting and search engine optimization. IRDI could lead to a single, unified index for academic institutional research data. A unified data index would lead to three key impacts: increasing discovery, reuse, and citation of open research data; reinforcing the idea that research data is a legitimate scholarly product; and promoting community-wide systems that require less resource expenditure.

They noted that getting a research data repository up and running is hard and expensive.

Their solution:

Deposit data elsewhere in discipline repositories
Have a local IRDI - Institutional Research Data Index

This is similar to the ReDBox approach in that it is highly distributed and it contains an IRDI. The Montana team have soon-to-be-released code that helps find data that's residing out there on the web, eg in Figshare, which we need to look in to.

Another thing we should explore ReDBOX is more about how data is stored and secured. We are working with a product, the DELL/EMC Isilon which has a lot of features in this area, but I plan to look at the Edinburgh & Manchester DataVault project as well Sustaining the momentum, moving the DataVault project to a service Claire Knowles presented with with Mary McDerby, Robin Rice, Thomas Higgins

DataVault does encrypted multi-site storage. 3 copies. One on site, one outer Edinburgh on UK cloud. They use chunking to reduce risk around encryption - lose less of a file if there's a problem.

Some highlights

An insight from Esme Cowles in Valkyrie: Reimagining the Samvera Community which is looking at adding swappable back ends to the Samvera platform (so you can use something other than Fedora to store your stuff).

Lesson from Islandora - don't fight your host platform.

Speaking of Islandora, in Relational Databases as Repository Objects Alexander Garnett showed off a plugin that gives live access to a SQL database in your Islandora repository; it spins up a Docker container on demand. I asked Alex on The Twitter if he'd seen Datasette but he said that SQLite does not scale well in his experience.

Thomas Morrell from Caltech mentioned a few interesting things in Positioning a repository as campus research infrastructure:

Live visualization from a data reposioty
Demoed fetching data from a live repository using a Jupyter notebook.
Harvesting metadata out of git repositories using the codemeta standard. Codemeta metadata is very close in design to our DataCrate. this could be straight out of DataCrate:

    {"author":
      {
      "@id":"http://orcid.org/0000-0003-0077-4738",
      "@type":"Person",
      "email":"slaughter@nceas.ucsb.edu",
      "givenName":"Peter",
          "familyName: "Slaughter"
      }
    }

Joshua A. Westgard from the University of Maryland Libraries presented a Python Library for the Fedora API, which would have been of interest to us at UTS if we'd gone ahead with our planned Fedora 4 design for data management instead of joining forces with QCIF in the new MongoDB based ReDBox.

Chris Diaz from Northwestern University talked about making Static websites over the top of a repo in Jekyll and Institutional Repositories, looks like a great, sustainable alternative to Omeka exhibitions worth considering in some situations.

And the winners are...

I thought I'd pick out a few of the best quotes and one-liners from my notes.

Best quote: Robin Dean

Fractional agile: how to do iterative development when it’s only part of your job

Abstract

Agile development methods such as scrum strongly recommend a dedicated team for software development, meaning that all team members work solely on one project at a time. This is not feasible in many of the academic and open-source community environments where open repositories are developed. This presentation will encourage developers, managers, and open-source contributors to find sustainable ways to participate in agile development even if they can’t dedicate themselves full time to a project. Drawing from my experience as a scrum master on a digital repository team, I will share practical advice on setting expectations, communicating about availability, and maintaining a sustainable pace on a scrum team composed entirely of part-time members.

And the quote:

THERE IS NO SUCH THING AS AN EXPECTATION THAT IS TOO CLEAR

Ta Robin, I've been using this at work.
Best catchphrase: Robert S. Doiel, Thomas Morrell

Building software at the edges of heterogeneous repositories Caltech Library, United States of America

Abstract

Caltech Library has a heterogeneous mix of repository systems (e.g. EPrints hosts CaltechAUTHORS and CaltechTHESIS, while CaltechDATA is based on Invenio). Caltech Library has changed its focus from developing in the specific repository system to one of development at the edges leveraging web APIs. This has allowed us to not only repurpose content but start working at collection level curation by integrating external data sources like ORCID, CrossRef, FundRef and DataCite. The philosophy we have evolved is to work from copies of the data in JSON form using an Open Source tool Caltech Library created called dataset as well as additional Open Source tools in a project called datatools. These command line tools are written in Go but can be easily used from more popular languages like Python. This talk will introducing these tools and demonstrate their usage via Python.

And the phrase:

CONTINUOUS MIGRATION

I liked this phrase because we're migrating data at the moment between RedBox 1 and ReDBox 2 and it's causing my colleague Michael Lynch visible physical and psychic pain. Robert's articulation of the idea that data migration should be normal, and done constantly so it's easy and considered business as usual resonated. But now that I think about it - what we're really aiming for is to get the core data to stay put and not to move - as per the aims of the Oxford Common Filesystem Layout I quoted above. The systems that index, access and analyse the data, and trade metadata are the ones involved in continuous migration.

I am please to report that Robert has recently been in contact with me to talk about aligning their dataset project with DataCrate, this is a step towards DataCrate being able to look inside data files and do stuff like describe column-headers in tabular data, rather than just describing the files.
Best "Lesson Learned": Lars Holm Nielsen, Alexandros Ioannidis, Krzysztof Nowak, Jose Benito Gonzalez Lopez from CERN

File loss: hits and near misses

Abstract

Repositories increasingly depend on external cloud storage or other complex distributed systems in order to satisfy ever-growing needs for storing larger data volumes. The cloud system helps repository manages store terabytes and petabytes of data, and often simplifies the file management in the underlying repository software. We trust these systems to store our files, yet, often we lack understanding of the operation and internals of these systems and how they can fail. This talk will present two file loss incidents on Zenodo, uncovering some ways these distributed systems can fail. One incident was caused by a coincidence of two software bugs in independent systems (the hit), and a second incident was caused by a human operational mistake in the cloud storage system (the near miss).

Great talk about when things go wrong. We don't hear enough of these stories.

The lesson?

LOG EVERYTHING

The venue etc

Often the OR conference is at a conference centre when in North America but this one was at the uni, in a lovely part of Bozeman, walkable from Downtown, through a leafy suburb of early twentieth century houses. The venue was comfy, the food was fine, and there was a cash bar at the dinner, though with a distinct lack of options for people who didn't want alcohol, and a decent band, Little Jane and the pistol whips. Gotta love that gun culture. Downtown Bozeman is great, plenty of good food options including Bison from Ted Turner's ranch. Ted not only revolutionised TV news by inventing CNN, he is a largest private owner of Bison in the world according to our server. He's even involved in research...

Q. What did Ted Turner discover when he let 200 Bison loose in the top paddock for 1 year?

A. The bisontennial.

Open Repositories 2018 Presentation: ReDBox 2.0 / Provisioner

2018-07-06T00:00:00+10:00

This is a presentation that Gavin Kennedy and I gave at Open Repositories 2018 in Bozeman Montana.

I am posting this on the UTS eResearch website and on my own site.

Gavin Kennedy, QCIF Peter Sefton, University of Technology Sydney ' title='ReDBox 2.0 / Provisioner

Gavin Kennedy, QCIF Peter Sefton, University of Technology Sydney ' border='1' width='85%'/>

Notes - Slide 1

The first part of this presentation was narrated by Gavin.

Notes - Slide 2

ReDBox is a Research Data Management Platform. It assists researchers and institutions to plan, create and publish their research data assets.

It is proudly open source.
It is managed and supported by the QCIF Engineering Services team.
It is fully customisable.
It is an integration platform, it will join your platforms together. It is interoperable, your platforms will happily exchange information with ReDBox.
It is in the cloud: mostly, we are progressively increasing our cloud options.
But most important is that it is community driven, the work we do to continuously improve ReDBox is determined, prioritised and often contributed by our community.

Notes - Slide 3

ReDBox has a rich history, with ongoing support from the Australian eResearch sector. Originally ReDBox was created on top of the Fascinator platform by USQ back in 2010 and taken up by successive institutions under the ANDS Metadata Stores funding program. In 2012 QCIF took over the management of ReDBox, giving it a stable non-partisan home. 2015 saw our first cloud version in ReDBox Lite, last year we produced our first SaaS version with the RAPportal, and this year we are releasing ReDBox 2.

Notes - Slide 4

This slide shows the institutions we know are using ReDBOX. The ones above the line have support contract with QCIF which help to pay for maintenance and enhancement of the software.

' title='ReDBox 1.x Institutional Data Catalogue ++ Metadata Store Mint Name Space Authority DMPt Curator Harvester Transformers API OAI/PMH Publisher

' border='1' width='85%'/>

Notes - Slide 5

At its core ReDBox is a customisable research data registry, providing forms, workflows and system integrations to assist institutions in creating and managing the metadata describing research data collections. It includes the innovative Mint platform to provide a name-authority lookup service for researcher information and project details, as well as commonly used research classifications, such as FOR codes. Mint helps to create high quality linked-data metadata, by supplying URIs as metadata values rather than strings which can be error-prone and ambiguous.

It incorporates a Data Management Planning tool, curation functions and a flexible harvester tool to pull data in from external platforms. It is a schemaless forms driven platform, incorporating configurable transformers to generate metadata from a standard schema, like RIF/CS or DataCite Citation format.

ReDBox includes a full set of APIs and provides an OAI-PMH interface for metadata harvesting by repositories including Research Data Australia.

' title='ReDBox 2.0 Technology refresh Single web application framework (Sails.js) Easy to configure forms New object model (JSON-LD in MongoDB) Modular/plug-in architecture Multi-institutional RBaaS (e.g. RAPortal.org.au) DLC (Data LifeCycle) support via RAiDs maDMPs Workspace concept to manage resources and datasets Flexible storage provisioning with the UTS Provisioner Framework Dataset harvesting, archiving (OCFL) and publishing Comprehensive search/discovery tool

' border='1' width='85%'/>

Notes - Slide 6

In late 2017 QCIF embarked upon a complete refresh of the ReDBox technology stack, assisted by RDS and UTS. We threw out the clunky front end tech stack and used Angular.js, a javascript framework that hasn’t gone off overnight. We have created a form configuration process to let you create your own forms and style them using bootstrap. It is easy to extend functionality using a plugin architecture. We can deploy it so multiple institutions to run off a hosted version of ReDBox, our ReDBox as a Service offering. The DMP tool now supports Machine Actionable DMPs. We have introduced a Workspace concept for managing resources within ReDBox. And with UTS we are implementing provisioning, data harvesting and rewriting the search/discovery tool.

Notes - Slide 7

And this is where the Data Life Cycle is supported. It allows users to plan for their data, acquire the resources for storing their data, harvest file level data and metadata back into ReDBox, curate it, archive it, publish it and make it discoverable.

Notes - Slide 8

Mercifully for you, this is not a demo. This is just a screen shot of a full blown DMP, with the DMP components organised in a tabular form. At this point I will remind everyone that it is fully configurable and that the forms can be as minimal or extensive as you require.

' title='RAP www.raportal.org.au Configurable SaaS Platform Proto-DMP RAiD Support in ReDBox Institutional Views

' border='1' width='85%'/>

Notes - Slide 9

An example of the minimum form is the Research Activity Portal (RAP). The RAP is our first configurable SaaS platform. It is a proto-DMP designed for users to register their research activity in order to get a RAiD, which is a Research Activity ID. It is available nationally in Australia and supports institutional views, so an individual institution can have a RAP themed to their institution with customised forms.

' title='Provisioner - principles

' border='1' width='85%'/>

Notes - Slide 10

At this point Petie took over the talking.

The next few slides go through some of the principles behind the service-provisioning aspect of ReDBOX.

Notes - Slide 11

We try to appeal to Researcher’s better selves: who doesn’t want to do high integrity research that has high impact?

Knowing where your data are is obviously key to Research Integrity, and is mandated by research norms, codes of practice and by funders. The provisioner helps with this; because provisioning can be invoked from a Research Data Management Plan (RDMP), the provisioned research space - we call them workspaces - has a bit of metadata that points back to the RDMP, which carries details about authorship, ownership of rights, retention requirements etc.

We also try to improve research impact - by making data available with as much provenance as possible to encourage re-use.

Notes - Slide 12

“An army of bots” is not really a principle, the principle is that everything possible should be done to automatically capture research data where it is and link it back to its custodian, and to metadata about the context in which the data were collected, curated or created. In the eResearch team at UTS we have long experience with customer requests such as “can we make sure that every student has a file-share to which the supervising panel also has read access even as the supervisors change” or “can I have an eNotebook for every PhD candidate in my department”. The provisioner will help with this kind of automation and more; we want to do things like report on how many electronic notebooks or git repositories are owned by the staff and students in a cohort that are NOT linked to an RDMP.

' title='Small Pieces, loosely joined Avoid point to point integration, use standards for integration

' border='1' width='85%'/>

Notes - Slide 13

When we started work on what became the Provisioner module of ReDBOX we were looking at how to integrate data applications that house research workspaces. For example how might we assist researchers in moving - maybe by reference rather than copying - data from a project in a microsocope-image database to the storage service in a visualization facility so the developers there can work on analytics and visualizations, and then archive the original data, code and outputs of the viz process? When we considered the scope of this and that we might be supporting dozens of applications that provide workspaces over the next few years, and working more loosely with hundreds more we realiased that we didn’t want to be doing n * n integrations, where n is the number of research apps we support. We looked to standards for managing data.

DataCrate Packaging ' border='1' width='85%'/>

Notes - Slide 14

One of the key standards was a way of moving “packages” of research data, where a package might mean a zip file, a manifest of data-by-reference or a directory of well-described data on a file share. From this search was born DataCrate - a specification which links together several existing standards into what we think is a best-practice generic way to ship and display research data.

This screenshot is of a the index.html file from a DataCrate which you can take a look at online or download as a zip file.

DataCrate Packaging ' border='1' width='85%'/>

Notes - Slide 15

You can read more about the spec at github and in the presentation I (Petie) gave at OR2018.

Summary: It uses BagIt for organizing files with checksums, JSON-LD for high-fidelity extensible metadata (linked data in JSON format), uses the schema.org vocabulary where possible for general metadata (dates, places, people) and specifies where to look for terms that are not covered by schema.org.

Notes - Slide 16

This diagram is a highly abstracted view of the UTS architecture for research-data management. Researchers are shown in many places in this diagram to emphasise, that while they can use the Stash (ReDBox) portal to edit Research Data Management Plans, and describe data sets, them can ALSO continue to use research workspaces independently. We aim to have an army of ‘provisioner bots’ working to help keep track of this.

Notes - Slide 17

This is a mock-up of the Redbox 2.0 functionality showing the way researchers will be able to manage the research process, creating data management plans and requesting workspaces, as well as identifying existing workspaces and linking to them. The right-hand screenshot shows how gitlab projects can be linked to a Research Data Management Plan as workspaces.

' border='1' width='85%'/>

Notes - Slide 18

This animation shows how a researcher will create a new workspace - in this case a gitlab repository. The Provisioner uses the gitlab API to connect as the user, create a new project/workspace, and leaves behind a calling card which links the project back to the RDMP. Later, the researcher will be able to archive and maybe publish this data via Provisioner, and it will know where to find the data.

Notes - Slide 19

This diagram shows how Research apps will be loosely coupled with Stash (the UTS name for our data management system that runs ReDBOX).

Notes - Slide 20

This diagram shows the technical architecture of the system. The “Repository” is actually several loosely coupled sub systems. The UTS implementation of ReDBox 2 will be a kind of “deconstructed” repository with the functions we see in more monolithic software such as DSPace or Samvera residing in different places. For example ReDBox is about data and workspace management, but the Data resides in research apps and in a static file-based archive, and the public-facing discovery service will be a separate application (which we’re working on at the moment).

Notes - Slide 21

This slide lists some of the future possibilities for ReDBox - we’re particularly interested in its potential at a national or consortial level.

Notes - Slide 22

Finally (from Gavin) some words on sustainability.

ReDBox is free open source software and you can download and run it today. But

QCIF is a not for profit organisation that can’t afford to cover the costs of ReDBox and the NCRIS grants are a distant memory.
So we have multiple models for subscribing to a support service. This covers our maintenance costs.
We fund development through projects like our collaboration with UTS.
We offer increasingly commercial services like managed hosting of ReDBox.
We are rolling out the Software as a Service (SaaS) version with a transactional charging model, subject ot feedback, next year.
We depend on our community and free and open communication with our community, so we know what each are doing.
And we depend on our community to drive the direction of ReDBox.

' title='More www.redboxresearchdata.com.au

' border='1' width='85%'/>

Notes - Slide 23

If you would like to know more please check out our website.

End-to-End Research Data Management for the Responsible Conduct of Research at the University of Technology Sydney

2018-07-04T00:00:00+10:00

This presentation was written by Louise Wheeler, Sharyn Wise and me for the Asia Pacific Research Integrity 2018 meeting in Taiwan, Feb 2018 it was scripted and delivered by Louise, who is the UTS Manager, Research Integrity and Research Program and Sharyn works in the eResearch team with me. This is good introduction to the work we've been doing on the UTS provisioner project from Louise's Research Integrity (RI) perspective. There's not much technical detail in this talk about the open source ReDBox platform on which our data management system, Stash, is built. I'll post more soon about that.

Thanks also to Chris Evenhuis for some slide design.

I'm posting this both on the eResearch blog and at my site.

Notes

Today I’ll be presenting the UTS approach to Research Data Management (RDM), and highlighting how the development of a practical and comprehensive solution can enable researchers to meet their RI obligations

I’ll run through a brief history that lead to our approach, and then I will present an overview of the solution we are implementing.

Formal RI training may occur early in a researchers career, but otherwise becomes assumed knowledge. Awareness of responsible research practices is not often actively supported or promoted throughout an academic's career, at least not consistently at the institutional level. At UTS we have pockets of opportunities for promoting research integrity principles, and one such pocket is the eResearch team – who have regular face-time with researchers when supporting them with tools and practices. We are using these opportunities to build a comprehensive framework that will enable researchers to meet RI principles, through improved systems and practices.

Image Source: www.arc.gov.au, http://www.dijifi.com/service/data-management/ ' border='1' width='85%'/>

Notes

Australia’s current code has been in place since 2007. It is a relatively prescriptive document that guides institutions and researchers as to their responsibilities in meeting integrity principles, including management of research data.

Despite these guidelines, at that time there was no defined or coordinated approach to research data management at UTS. Any management relied on the best endeavours of individual researchers to protect their data. They continued to store their data on file-servers; in drawers, in unlabelled hard drives, in their garages etc.

Image Source: www.arc.gov.au, http://www.dijifi.com/service/data-management/ ' border='1' width='85%'/>

Notes

It was not until 2013 that national infrastructure funding allowed Australian Universities to develop Research Data catalogues. For the first time researchers were encouraged to think of their data as an asset to be managed, as opposed to simply kept. At the same time, national requirements to include RDM plans in proposals to our major funding bodies, gave universities a bigger driver for changing culture.

At UTS we began to govern research data at the Policy level, which required researchers to plan their data management, At the systems level, because of the new focus on RDM requirements, we had the necessary senior-level investment to develop our research data catalogue (which we call Stash).

Take up of Stash over the next two years was very slow. With minimal resources, the strategy for promoting Stash focused on educating graduate research students and thereby indirectly, their supervisors. It was difficult to persuade researchers that sharing their data was a good idea, and that storage should be on university owned and controlled systems.

And, since the ANDS funding rules didn’t allow for development of data repositories, our catalogues weren’t integrated with our storage solutions. It was clear at that point that researchers needed an integrated data management solution.

Image Source: www.arc.gov.au, http://www.dijifi.com/service/data-management/ ' border='1' width='85%'/>

Notes

In 2016 efforts around RI and RDM began to align. We rolled out a new Research Integrity Framework while the eResearch team produced a Strategy and roadmap that explicitly linked data management to Integrity requirements and during 2017, our research policy environment was redrafted in the same vein.

Image Source: https://las.inf.ethz.ch/research/large-scale-machine-learning Image Source: www.arc.gov.au, http://www.dijifi.com/service/data-management/ ' border='1' width='85%'/>

Notes

Collectively, these efforts enabled us to put the pieces in place for an end-to-end data management solution and the Provisioner project was born.

Provisioner

Provisioner Picture credit: Gerrad Barthelot, Technical Architect, IT Infrastructure UTS ' border='1' width='85%'/>

Notes

This is Provisioner. It looks complex, but I will walk you through it using a couple of examples that demonstrate how it supports both RDM and research integrity.

Provisioner

Provisioner Research Data Mgmt Plan Research Data Catalogue

Picture credit: Gerrad Barthelot, Technical Architect, IT Infrastructure UTS ' border='1' width='85%'/>

Notes

A researcher (top centre) accesses the data management catalogue “Stash”, which has three parts: (1) Create a data management plan (as required by our policy environment), to describe how their data will be collected, analysed, stored and accessed. (2) Access research data catalogue listing where archived data sets can be found

Provisioner

Research Data Mgmt Plan Research Data Catalogue Picture credit: Gerrad Barthelot, Technical Architect, IT Infrastructure UTS ' border='1' width='85%'/>

Notes

And from the plan, they (will by the end of 2018) have access to (3) the innovative provisioning tool which allows researchers to provision workspaces such as file-shares, electronic notebooks, and repositories for programming code,

Provisioner

Provisioner Research Data Mgmt Plan Research Data Catalogue Picture credit: Gerrad Barthelot, Technical Architect, IT Infrastructure UTS ' border='1' width='85%'/>

Notes

They can request and link to data storage, and at the end of the project archived and publish their datasets.

Provisioner

Picture credit: Gerrad Barthelot, Technical Architect, IT Infrastructure UTS ' border='1' width='85%'/>

Notes

The Provisioner will also have a range of reporting tools and automated ‘bots’ than generate useful reports for the Research Integrity team – identifying ‘orphaned’ workspaces, or auto-creating data management plans for researcher or research students.

Information is pulled in from our existing databases to populate the RDMPs, again saving researchers time.

Provisioner

Provisioner Picture credit: Gerrad Barthelot, Technical Architect, IT Infrastructure UTS ' border='1' width='85%'/>

Notes

We had a recent example in a science lab where some code from one project was modified for use in a new project. Most researchers are not aware that code is considered research data that must be retained, particularly in the sciences where they are not trained in IT. So no copies of the code were retained, and that data is now lost. In the Provisioner model, Gitlab solves this problem without the researcher having to stop and think about their research integrity obligations by saving each version the code and creating a revision history.

By introducing Provisioner, we are not implying that researchers can remain blissfully unaware of their responsibilities, but we are supporting them by providing a positive and streamlined experience, with the added comfort of understanding that it comprehensively addresses responsible research practices. We will continue to raise awareness by linking integrity training to our practical training around data mgmt, and other element of the project lifecycle.

We’ve found that bringing researchers into contact with our eResearch experts is an effective way of supporting and building an integrity culture. Our researchers tend to prefer practice-based over principles-based education: so we are aiming to build the principles into our workshops/clinics (e.g. Research ethics practices, Research data management practices, Publications workshops that focus on open access and reproducibility

Provisioner

Provisioner Picture credit: Gerrad Barthelot, Technical Architect, IT Infrastructure UTS ' border='1' width='85%'/>

Notes

Another example of how Provisioner addresses RCR is the incorporation of the LabArchives/ELN (Electronic Lab Notebook)(top left). As this model is being implemented uni-wide, it gives us an institutional advantage of oversight. eNotebooks developed at the lab or faculty level leads to inconsistent practices and means that data is out of the universities control.

Our policy dictates that rights in data are owned by the institution, which means we should have some control over how it is stored, accessed and reported, and also have the ability to interrogate the system, should any disputes arise. The LabArchives tool ensures that data can not be manipulated or lost, can help to clarify IP and ownership issues, and also enables us to manage data access when researchers move institutions.

Provisioner

Provisioner Picture credit: Gerrad Barthelot, Technical Architect, IT Infrastructure UTS ' border='1' width='85%'/>

Notes

And finally, for researchers doing real-live imaging experiments, they could generate terabytes of data in one session, so Provisioner can provide access to large, online repositories such as Omero. This gives researchers comfort that all their experimental data can be kept, not just selected files.

Picture credit: Screen shot of Stash 2.0, UTS Research Data Catalogue ' border='1' width='85%'/>

Notes

Provisioner is still being developed throughout 2018. What we’ve learned so far is that:

Simply mandating new RDM requirements doesn’t work. Researchers are incentivised by integrated and discipline-relevant tools that save them time and give them reason to uphold good data management practices for more than just policy compliance.
Successful implementation requires stakeholder engagement and championship at senior levels - we are fortunate that our Associate Dean, Research (ADR) in Health has invested heavily in both our RI and RDM responsibilities and has implemented a cultural change in that Faculty, meaning that every project now has an RDMP. This is reflected in uptake of RDMPs across the institution in 2017, supported by the extensive cross-promotion of between Ethics, RI/RCR and RDM representatives at UTS.

' border='1' width='85%'/>

Notes

Early feedback has been positive with researchers keen to take up the components we have rolled out because they can see immediate value in adopting them, and we can already begin to see the effectiveness of the model by the levels of uptake of our LabArchives and GitLab since their implementation in 2017.

However, there is still a way to go. Data archiving and preservation to cater for longer retention periods such as those demanded by clinical trials are still ahead on the eResearch roadmap. Data management practice is improving but is by no means universal. We accept that culture change concerning RDM is a long term proposition; but it can also be a quick win for RCR by offering an opportunity to promote research integrity in practical ways. Meanwhile, we take a risk management approach, focusing on sensitive and personal data and grants where the funding bodies require RCR and research data management.

Acknowledgements STASH and Provisioner development is supported by the Australian National Data Service (ANDS) through the Australian Government’s National Collaborative Research Infrastructure Strategy Program and the Education Investment Fund (EIF) Super Science Initiative. STASH is based on ReDBox, developed by Queensland Cyber Infrastructure Foundation Ltd (QCIF) Louise Wheeler – Manager, Research Integrity louise.wheeler@uts.edu.au Sharyn Wise – eResearch Analyst sharyn.wise@uts.edu.au Peter Sefton – Manager, eResearch Support peter.sefton@uts.edu.au Authors ' border='1' width='85%'/>

Notes

Introducing Provisioner

2018-04-05T00:00:00+10:00

This post is an introduction to the Provisioner, an open framework for research data management which we're developing in collaboration with the Queensland Cyber Infrastructure Foundation, QCIF and the Australian National Data Service, ANDS.

Provisioner grew out of a project funded by the UTS IT Capital Managemement Program, which is, confusingly, also called Provisioner. In this post, I'll use "Provisioner" to refer to the framework and software, not the project as a whole.

The original goal of the project was to provide a data pipeline which would manage the transfer of microscopy data from the Microbial Imaging Facility, MIF to the visualisation environment at the Data Arena, with some analysis and processing on the way.

During the initial scoping and design work for the project, the middleware system to support the transfer of data from MIF to the Data Arena started to look like a desirable technology in itself. This middleware is what became the Provisioner. The final design for Provisioner is an attempt to address two problems -

how do we support end-to-end data management without taking researchers away from the tools they use for research or requiring them to manually enter administrative data?
how do we integrate data management into research tools without having to customise a growing list of software platforms?

The best introduction to Provisioner is to describe how most researchers will interact with it.

Service catalogue

Stash is the UTS research data catalogue, allowing researchers to create RDMPs (research data management plans) and describe datasets. Stash is currently implemented in ReDBox 1.9, an open source platform maintained by QCIF. As part of this project, QCIF are reimplementing ReDBox using modern web frameworks.

The new version of ReDBox will include a service catalogue, which will allow researchers to request a range of IT services for use with a project with an RDPM.

This diagram shows where we'd like to be in a few years with Provisioner managing research data across

file storage on local or cloud services
research code repositories in GitLab
dedicated microscopy management from OMERO
electronic lab notebooks from labarchives
secure data collection surveys from REDCap
and other services that will be requested by our researchers

The current project is aiming to provide connectors to GitLab, OMERO and labarchives.

The Provisioner terminology for these services is research apps (or just apps), and a directory, site or project belonging to a researcher in a particular app is referred to as a workspace.

When a researcher requests a workspace via the service catalogue, Stash creates a workspace record in its own database which is linked to the user and RDMP. It then calls the Provisioner API to request a new workspace. The Provisioner API translates this request into the actual operations needed to provision the workspace in that particular research app, and writes a metadata file into the workspace which links it to the new workspace record, the researcher and the RDMP.

The Stash interface then reports back that the workspace has been created, and gives the research a URI and instructions for how to access it.

This has given the researcher a resource which they can start using, and has ensured via the metadata file that the resource can be linked back to the RDMP and project which govern it.

Metadata

Once a research has used it to get access to a research app, Provisioner will get out of their way and let them use the app's native interface. Even this minimal use-case will be a big step forward in terms of research data management: we'll have a system which can make sure that new research IT storage and workspaces contain documentation, in the form of the Provisioner metadata, which describes the data and links back to the RDMP and researcher who are responsible for it.

The exact spec of the Provisioner metadata is evolving as we develop the system. We're going to use DataCrate, an evolving standard for packaging and documenting research data. DataCrate provides an HTML version which can be used as a landing page or manifest when a dataset is published - here's a sample dataset of LIDAR data from the Wombeyan caves.

The same data is also provided as a JSON-LD document which is easily machine-readable and follows linked open data principles.

DataCrate is flexible about the level of detail it can provide: it's possible but not necessary to go down to the level of individual files. This means that early in the research life-cycle, a workspace can have broad, high-level descriptions, which can be filled in as required when a dataset is moved into the publication workflow.

Managing workspaces

The workspace interface in Stash will allow researchers to carry out high-level management tasks. By "high-level", we mean operations which make sense when applied to a workspace in any research app, rather than those which only apply to, for example, microscopy, or file storage.

The range of high-level operations will include:

sharing a workspace with other researchers
setting a workspace to be immutable (read-only)
making a workspace from one app available in a different research app

The Provisioner API is being designed to provide a common interface for these operations. Each research app will have an adaptor which translates these API calls into the native API for that app.

The third operation listed above, making a workspace from one app available in another, is how Provisioner will satisfy the initial requirement: a data pipeline from MIF to the Data Arena. It also has the potential to support many more useful processes, such as data archiving and publication. I'll go into more detail about this in the next blog post.

Upcoming Training Courses in 2018 Have been Updated

2018-02-28T00:00:00+11:00

Upcoming Training Courses in 2018 Have been Updated

Upcoming training courses are updated regularly. The latest update has been done on 28 Feb 2018. Courses after April are not open for registration yet, please keep an eye on our training page.

DataCrate: Formalising ways of packaging research data for re-use and dissemination

2018-01-30T00:00:00+11:00

By Peter Sefton

A version of this post is also available at my website.

This is a presentation I gave at eResearch Australasia 2017-10-18 about the new Draft (v0.1) Data Crate Specification for data packaging I've just completed, with lots of help from others (credits at the end).

BACKGROUND

In 2013 Peter Sefton and Peter Bugeia presented at eResearch Australasia on a format for packaging research data(1), using standards based metadata, with one innovative feature – instead of including metadata in a machine readable format only, each data package came with an HTML file that contained both human and machine readable metadata, via RDFa, which allows semantic assertions to be embedded in a web page.

Variations of this technique have been included in various software products over the last few years, but the there was no agreed standard on which vocabularies to use for metadata, or specification of how the files fitted together.

THE PRESENTATION

This presentation will describe work in progress on the DataCrate specification(2), illustrated with examples, including a tool to create DataCrate. We will also discuss other work in this area, including Research Object Bundles (3) and DataConservency(4) packaging.

We will be seeking feedback from the community on this work should it continue? Is it useful? Who can help out? The DataCrate spec:

Has both human and machine readable metadata at a package (data set/collection) level as well as at a file level
Allows for and encourages inclusion of contextual metadata such as descriptions of organisations, facilities, experiments and people linked to files with meaningful relationships (eg to say a file was created by a particular machine, as part of a particular experiment, at an organisation).
Is a BagIt profile(5). BagIt(6) is a simple packaging standard for file-based data.
Has a README.html tag file at the root with bagit-style metadata about the distribution (contact details etc) with a link to;
a CATALOG.html file in RDFa, using schema.org metadata inside the payload (data) dir with detailed information about the files in the package, and a redundant CATALOG.json in JSON-LD format
Is extensible easily as it is based on RDF.

REFERENCES

Sefton P, Bugeia P. Introducing next year’s model, the data-crate; applied standards for data-set packaging. In: eResearch Australasia 2013 [Internet]. Brisbane, Australia; 2013. Available from: http://eresearchau.files.wordpress.com/2013/08/eresau2013_submission_57.pdf

datacrate: Bagit-based data packaging specification for dissemination of research data with useful human and machine readable metadata: “Make Data Crate Again!” [Internet]. UTS-eResearch; 2017 [cited 2017 Jun 29]. Available from: https://github.com/UTS-eResearch/datacrate

Research Object Bundle [Internet]. [cited 2017 Jun 16]. Available from: https://researchobject.github.io/specifications/bundle/

Data Conservancy Packaging Specification Home [Internet]. [cited 2017 Jun 29]. Available from: http://dataconservancy.github.io/dc-packaging-spec/dc-packaging-spec-1.0.html

Ruest N. BagIt Profiles Specification [Internet]. 2017 Jun. Available from: https://github.com/ruebot/bagit-profiles

Kunze J, Boyko A, Vargas B, Madden L, Littman J. The BagIt File Packaging Format (V0.97) [Internet]. [cited 2013 Mar 1]. Available from: http://tools.ietf.org/html/draft-kunze-bagit-06

This is a presentation I gave at eResearch Australasia 2017-10-18.

Peter Bugeia and I talked about this 4 years ago. This year I got around to leading the effort to standardising what we did back then.

This presentation is structured as a story.

Back in June Cameron Neylon was annoyed

When I saw this cry for help I contacted Cameron and offered to work with him.

2017 eResearch Strategy update

2018-01-04T00:00:00+11:00

We have just completed our first annual review of progress on the 2016-2020 eResearch strategy.

Upcoming Training Courses Have been Updated

2017-09-15T00:00:00+10:00

Upcoming Training Courses Have been Updated

Upcoming training courses are updated regularly. Please keep an eye on our training page. The latest update has been done on 15 September 2017.

Announcing Ozmeka - Linked Data enhancements for Omeka

2016-03-01T00:00:00+11:00

In collaboration with Western Sydney University (formerly UWS), the eResearch team here at UTS has been working on some enhancements to the Omeka repository tool. We presented a general paper about this at Open Repositories 2015, but in this (very late!) post we'd like to talk a bit about the software.

There are a couple of changes we made that we think might be good additions to the Omeka v2 core. I called in to visit the Omeka team at George Mason University in June 2015 to demo this work, but we have yet to reach out to the broader Omeka community.

Please let us know what you think. We'll try to have that discussion over in the Omeka forums or on Twitter, rather than here, though.

What is ozmeka?

Ozmeka is a set of plugins, or forks of plugins, to make Omeka into more of a linked data platform. That is, to enable Omeka repositories to embrace linked data principles, the first two of which are:

Use URIs as names for things.

Use HTTP URIs so that people can look up those names.

The new version of Omeka, known as Omeka S, promises to fix a lot of the issues we were targeting with our work on Ozmeka, but it is not ready for production use yet.

Where can I get it?

Everything we're talking about here is available on Github, under the Ozmeka project. There is a Github Ozmeka Repository where we did some lightweight project management, with milestones and issues, but the plugins are all in individual repositories. We'll talk about them below.

What specifically did you produce?

We made changes, which we hope improved a bunch of Omeka plugins and produced some scripts for pushing data into Omeka.

Item Relations

The Omeka Item Relations plugin already existed, and it was one of the main reasons we chose Omeka, as it allowed not only repository items to have URIs but for items to be related to each other using terms from standard vocabuaries. This meant that instead of having metadata like:

This item <dc:Creator> "Some name"

You can have:

This item <dc:Creator> #21

Where #21 is a reference to another item in Omeka. This is a huge step-up in metadata quality over using strings as values.

What did we do:

Added a lookup to item relations, so you can find the thing you're trying to relate to. Previously you had to know the ID. Yes, the user interface could be improved but it works efficiently.
Added Item Relations to the API, so that external code can relate items. This is used in some of the utilities.
Added English glosses to relations so they make sense when displayed on the page. For example, the creator relation from the Dublin Core metadata standard is ambiguous. Does This item Creator Somebody mean that somebody created the item or somebody was created by the item. We added the ability for relations to be displayed like: This item was created by Somebody.
Added the ability to create new Item Relations vocabularies (if only via the API).

These all seem like good candidates to roll into the ItemRelations plugin.

Added URIs to the core data model / worked on lookups

One of the limitations of Omeka v 2.x is that it only accepts simple text values for metadata. Omeka-s fixes this, and metadata values can be text, a URI or a reference to another item. After much soul searching, Thom added a new database field to allow any metadata item to have a URI, and linked this into some new work on external lookups (presently SCOT (the (Australian) Schools Online Thesaurus), Geonames and the preexisting US Library of Congress lookup), so that we can let people select metadata values from external services, or input a URI as a value. Given that this is the approach taken in Omeka-s, this looks like an obvious thing to retro-fit to Omeka v2.x.

Sundry

We also have a simple modal dialogue to ensure visitors accept Terms and Conditions, an extra option (filter by Subject) in Search and a light plugin which renders attached HTML files inline in an item.
The Seasons theme has also been slightly modified to present linked images on a page with their metadata instead of raw.

Auto-ozmeka making it easy to deploy a new server

Auto-ozmeka is a collection of scripts that uses Git, Vagrant, Ansible and Virtualbox to create a virtual machine. The VM will have a fresh installation of Omeka and the suite of Ozmeka plugins ready to use. Solr search is preconfigured.

Enable the Ozmeka plugins you wish to use in the web GUI, add any others you may need, and you're ready to start developing your linked-data website.

In the near future we expect to be extending this script to provision production environments as well.

Utilities

In addition to the plugins we developed a few tools for using Omeka over the API. Work started on this at Western Sydney, and continued at UTS. The main tool, xlsx2omeka.py takes a spreadsheet of data and imports it into an Omeka repository, including the ability to create collections and upload multiple different types of item and multiple attachements per item.

Our spreadsheet uploader works, but it's clumsy, and I (Peter) have started working on a new approach that uses CSV files rather than spreadsheets, and does not require as much fiddling around, but this one is not yet very mature and is undocumented. The idea with the new approach is to allow the same data to be uploaded to multiple types of repository, we're also working on an uploader for Fedora 4 which is very much a work in progress.

Hacky Hour, HPC focus with special guest Dr Joachim Mai

2015-09-01T00:00:00+10:00

Attention HPC users! Are you ready to play in the big league?

We’ve noticed some sophisticated research compute happening on the UTS HPCCs, so this week Hacky Hour has a special guest for you.

Meet Raijin, the National Supercomputing facility (and star of ABC TV’s thriller “The Code”)!

Ok, so Raijin isn’t coming to us, but our guest, Dr Joachim Mai, can help bring you to Raijin.

If you are getting frustrated with capacity issues or queues it might be time to migrate to the big time.

Joachim will answer questions on HPC, parallel programming and provide advice to help you put together a merit application for NCI compute resources.

Not an HPC user? Not a problem ... come anyway!

The usual crew will be there. Hacky Hour is a weekly meetup where researchers can congregate to work on their research problems related to code, data, or digital tools in a social environment.

Bring along your challenges, digital tools or code to share. Have your technical research problems solved or find someone who can help you.

When & Where? Thursdays, 3pm-4pm at Penny Lane (the Cafe/bar in building 11).

Look for the group of people with laptops at the large table and join us for a drink.

HACKY HOUR : Technology support for researchers, by researchers (& eResearchers)

Hacky Hour 1.0

2015-08-20T00:00:00+10:00

Hacky Hour 1.0

Thank you to everyone who came to our first ever Hacky Hour at UTS. It was great to see so many people come along to get help with their problems, lend a hand to others or just catch up with other researchers or people on the eResearch team.

Here are just a few of the things people talked about:

Peter Sefton talked to people from the Microbial Imaging Facility about managing microscope image data using an application such as Omero.
Michael Lake talked to a researcher from the C3 Remote Sensing group about code execution performance (R vs Python), using profiling tools to optimise code, writing test suites and the benefits of using a version control system.
Sharyn Wise talked to A/Prof Cynthia Whitchurch and Dr Lynne Turnbull about IN Cell Miner.
Chris Bell, our current Server Administrator, caught up with researchers to go over outstanding ServiceConnect issues.
Michael Lynch helped a researcher from FEIT who's looking at measuring social media engagement. He wanted some help finding datasets and understanding the IP issues around collecting posts.
Darren Lee and Jared Berghold helped researcher Mingming Cheng with his data wrangling problem (see below). In the process, they introduced him to Python, the popular Python distribution called Anaconda and the excellent interactive programming tool called Jupyter (formerly known as IPython).

Problem Solved!

Mingming Cheng, a PhD researcher from the UTS Business School, came along with a problem he encountered while assisting his supervisor, Dr Deborah Edwards, on her project that examines the spatial behavior of tourists in NSW.

Mingming and Deborah are analysing public converations that have been extracted from TripAdvisor by the Advanced Analytics Institute at UTS and generating visualisations showing the interactions between forum participants.

Mingming needed the spreadsheet in the appropriate format for importing into the visualisation tool, Gephi.

The data was in this format:

But it needed to be in this format :

As you can see, the usernames in the author column had to be transposed onto a single row for each conversation thread. Conversation threads were identified using a unique index number. Mingming had over 65,000 rows in his spreadsheet, so doing this by hand was not a viable option.

Darren Lee from the UTS Data Arena came to the rescue, hacking up a Python script in the time it takes to drink a beer (or maybe two).

Here's what the script looked like:

import sys
import csv

with open('/Users/jared/Desktop/trial_mingming.csv') as csvfile:
    reader = csv.reader(csvfile)
    prevThreadNum = None

    with open ('/Users/jared/Desktop/trial_mingming_out.csv', 'w') as outfile:
        rowBuffer = []
    
        for row in reader:
            threadNum = row[0]
            if prevThreadNum == threadNum:
                rowBuffer.append(row[5])
            else:
                if len(rowBuffer):
                    #print rowBuffer[0] + "," + ";".join(rowBuffer[1:])
                    outfile.write(rowBuffer[0] + "," + ";".join(rowBuffer[1:]) + "\n")
                rowBuffer = []
                rowBuffer.append(row[0])
                rowBuffer.append(row[5])
                prevThreadNum = threadNum

The script takes a CSV (comma separated values) file as its input. The .csv file was generated by performing a File > Save As... in Excel and choosing .csv as the file format.

After reading in the CSV file, the Python script steps through the data, one row at a time. The first thing it does is stores the value from the first column (i.e. column '0') in a variable called threadNum: threadNum = row[0]. As you can see in the screenshot above, this first column is the index or conversation thread number.

The script then checks the index number to see if it has encountered that number before: prevThreadNum == threadNum. If it has, it continues to build up the list of usernames from the author column (i.e. the 6th column or column number '5' if you start counting from 0 like computers do).

If the script hasn't encountered that number before, then the script has come across a new conversation thread. It writes out the index number and a semi-colon separated list of all the usernames it has collected for that thread to the output .csv file: outfile.write(rowBuffer[0] + "," + ";".join(rowBuffer[1:]) + "\n").

As you can see in the screenshot above, the row in the new spreadsheet now has the index number in the first column, followed by a list of usernames separated by semi-colons (';') in the second column. It's this semi-colon separated list that is understood by Gephi.

We ran the script over Mingming's entire dataset and in less than a minute, he had the .csv file he needed to load into Gephi for visualising the data. Mingming was very grateful to have a solution to a problem he'd been grappling with for the last two months.

If you would like to know more about the script above, any of the tools mentioned (Python, Jupyter, Gephi) or any other digital research tool, then come along next week.

Thanks for reading and we look forward to seeing you at the next Hacky Hour!

Hacky Hour at UTS, Thursday arvo eResearch meetup / helpdesk starting August 13, 2015

2015-08-03T00:00:00+10:00

What is Hacky Hour?

Inspired by the ResBaz people at The University of Melbourne, the UTS eResearch team are launching Hacky Hour at UTS. Hacky Hour has been successfully replicated at other universities such as The University of Auckland and The University of British Columbia, so we’re excited to be bringing it to UTS!

Hacky Hour is a weekly meetup where researchers can congregate to work on their research problems related to code, data, or digital tools in a social environment.

There will be knowledgeable folk from the eResearch team there who can help you with your questions.

Why should I come to Hacky Hour?

Here are some reasons you might want to come to Hacky Hour:

You want someone to show you the basics in R.
You want to know the best way to show some spatial data on a map.
You recently attended an Intersect training course and want to get some more help using tool X.
Your honours student is pestering you to use you use version control and you want some help getting started.
You need to get a huge dataset from a colleague overseas and want to know the fastest way to do it.
You need a quick drink while you wait for your scripts to finish running...
Or you just want to hack away at your data-crunching scripts in some good company.

Even if you don’t have any problems you want solved, come along to help solve someone else’s. Help a colleague overcome their fears of using Git or show off a really useful tool you just discovered.

Over the coming weeks we’ll also be running some mini-workshops in things like Raspberry Pi and Jupyter. Stay tuned for more details.

I’m in. How do I get involved?

Just show up. If you have a laptop or tablet, you might want to bring it along.

Hacky Hour is a weekly get together on Thursdays at 3pm-4pm, at Penny Lane (the cafe/bar on the ground floor of building 11). The first Hacky Hour will be on Thursday 13 August. Come along and look for all the people with laptops.

We can’t wait to see you there!

Getting Started with Machine Learning in MATLAB

2015-07-29T00:00:00+10:00

Date: 25 August 2015

Time: 2.30pm – 3.30pm

Session Format: Presentation

Venue: CB11.06.408

Machine learning is being used to solve interesting problems in many fields: autonomous navigation, fraud detection, animal conservation, health care, energy forecasting. But many machine learning algorithms are complicated to implement, so how can you start using these algorithms on your applications?

In this session, Matt will show how the MATLAB® environment makes it easy to apply machine learning algorithms to your data. Attend this session to learn how to use functionality in Statistics and Machine Learning Toolbox™ and Neural Network Toolbox™ to:

Find natural patterns in your data
Build predictive models
Evaluate, interpret, and improve your models

Speaker Bio:

Matt Tearle is a training content developer for MathWorks USA. He develops instructional material on MATLAB for technical computing and data analysis. Prior to joining MathWorks, Matt taught applied mathematics and computing and the University of Colorado, Boulder, where he received his PhD in computational fluid dynamics in 2004. Matt has been using MATLAB since 1992, when he was an undergraduate at the University of Auckland. A true-born Cantabrian, he still supports the Crusaders.

RSVP: 5PM Tuesday August 18, 2015 to Matthew.Gaston@uts.edu.au