Panton Fellowships | OKF Open Science Working Group

You are browsing the archive for Panton Fellowships.

Congratulations to the Panton Fellows 2013-2014

Jenny Molloy - November 20, 2014 in Featured, Panton Fellowships

Samuel Moore, Rosie Graves and Peter Kraker are the 2013-2014 Open Knowledge Panton Fellows – tasked with experimenting, exploring and promoting open practises through their research over the last twelve months. They just posted their final reports so we’d like to heartily congratulate them on an excellent job and summarise their highlights for the Open Knowledge community.

Over the last two years the Panton Fellowships have supported five early career researchers to further the aims of the Panton Principles for Open Data in Science alongside their day to day research. The provision of additional funding goes some way towards this aim, but a key benefit of the programme is boosting the visibility of the Fellow’s work within the open community and introducing them to like-minded researchers and others within the Open Knowledge network.

On stage at the Open Science Panel Vienna (Photo by FWF/APA-Fotoservice/Thomas Preiss)

Peter Kraker (full report) is a postdoctoral researcher at the Know-Center in Graz and focused his fellowship work on two facets: open and transparent altmetrics and the promotion of open science in Austria and beyond. During his Fellowship Peter released the open source visualization Head Start, which gives scholars an overview of a research field based on relational information derived from altmetrics. Head Start continues to grow in functionality, has been incorporated into Open Knowledge Labs and is soon to be made available on a dedicated website funded by the fellowship.

Peter’s ultimate goal is to have an environment where everybody can create their own maps based on open knowledge and share them with the world. You are encouraged to contribute! In addition Peter has been highly active promoting open science, open access, altmetrics and reproducibility in Austria and beyond through events, presentations and prolific blogging, resulting in some great discussions generated on social media. He has also contributed to a German summary of open science activities every month and is currently involved in kick-starting a German-speaking open science group through the Austrian and German Open Knowledge local groups.

Rosie with an air quality monitor

Rosie Graves (full report) is a postdoctoral researcher at the University of Leicester and used her fellowship to develop an air quality sensing project in a primary school. This wasn’t always an easy ride, the sensor was successfully installed and an enthusiastic set of schoolhildren were on board, but a technical issue meant that data collection was cut short, so Rosie plans to resume in the New Year. Further collaborations on crowdsourcing and school involvement in atmospheric science were even more successful, including a pilot rain gauge measurement project and development of a cheap, open source air quality sensor which is sure to be of interest to other scientists around the Open Knowledge network and beyond. Rosie has enjoyed her Panton Fellowship year and was grateful for the support to pursue outreach and educational work:

“This fellowship has been a great opportunity for me to kick start a citizen science project … It also allowed me to attend conferences to discuss open data in air quality which received positive feedback from many colleagues.”

Samuel Moore (full report) is a doctoral researcher in the Centre for e-Research at King’s College London and successfully commissioned, crowdfunded and (nearly) published an open access book on open research data during his Panton Year: Issues in Open Research Data. The book is still in production but publication is due during November and we encourage everyone to take a look. This was a step towards addressing Sam’s assessment of the nascent state of open data in the humanities:

“The crucial thing now is to continue to reach out to the average researcher, highlighting the benefits that open data offers and ensuring that there is a stock of accessible resources offering practical advice to researchers on how to share their data.”

Another initiative Sam initiated during the fellowship was establishing the forthcoming Journal of Open Humanities Data with Ubiquity Press, which aims to incentivise data sharing through publication credit, which in turn makes data citable through usual academic paper citation practices. Ultimately the journal will help researchers share their data, recommending repositories and best practices in the field, and will also help them track the impact of their data through citations and altmetrics.

We believe it is vital to provide early career researchers with support to try new open approaches to scholarship and hope other organisations will take similar concrete steps to demonstrate the benefits and challenges of open science through positive action.

Finally, we’d like to thank the Computer and Communications Industry Association (CCIA) for their generosity in funding the 2013-14 Panton Fellowships.

No Comments »

Panton Fellowship- End of year post

Rosie Graves - November 11, 2014 in Featured, Panton Fellowships, Uncategorized

Well this last year seems to have flown by but I have to say I have really enjoyed my year as a Panton Fellow. As someone who had very little knowledge of all things open at the beginning of the year it’s been a great experience for me, meeting some interesting people, talking to scientists about open policies and learning more about the issues surrounding open data and open science. In this blog post I will summarise my work as a one of the 13/14 Panton Fellows and include some of the outputs of my project at the bottom.

I would like to say running my school based project has been easy and that I achieved everything I set out to do but unfortunately that’s not the case. As anyone who works with schools will know, it takes a lot of time and effort on both the parts of the visitor and the teachers and sometimes getting things to be implemented as quickly as you like is not always possible. This is not a criticism of teachers by the way, I see how overworked they are and any projects like this that they get involved with only add to their work load.

Introducing students to the project

So where to begin, well the aims of my project were pretty simple:

Install air quality sensors in primary schools.
Get students to collect data and work as the scientists.
Host data on a webpage to allow the local public to see what air quality is like in their area.
Increase knowledge of Open Data/ Open Science in the air quality field.

I think that it is safe to say that I definitely achieved elements of each of these aims as well as getting involved in lots of other things on the way.

If you read my June blog post then you will know that I eventually managed to get an air quality sensor and weather station into a school and I ran my first introductory session with the students. To say the students were engaged with the project was an understatement, they were so enthusiastic and their knowledge of all things air quality far surpassed my initial expectations. The sensor was installed in the school and successfully collected data from the 6^th-11^th of June. Then for some unknown reason it stopped working. This unfortunately coincided with my contact teacher leaving the school and the summer holidays starting. So an unfortunate series of events left me with five days of data for the last school year- not quite to plan. I am now back in touch with the new science coordinator at the school and we are hoping to start the project again before Christmas. In the meantime I have been making links to other schools and am looking to start work in a second school in the New Year.

So what else have I been up to this year? Well I have presented my Panton work at several conferences. The first one was in March at the Air Quality Conference in Garmisch-Parternkirchen and I received a great response from that- the blog post that I wrote about this conference is found here. I have also presented at several UK conferences, the NCEO/CEOI annual conference in Sheffield and the NCEO young scientist conference. The poster I presented at these conferences is found at the bottom of this blog.

Using electric cars to measure air quality in Leicester

Over the last year I have also continued to work with colleagues from RMetSoc and Manchester Met on citizen science projects to be run in school. We ran a successful pilot project last year with primary school students making rain gauges and then sending us daily rainfall measurements. We produced a paper outlining the results of this project for Weather in July this year. Over the next few months we plan to further this project by asking schools to buy a more reliable rain gauge and then increasing the size of the school network from the pilot project. Further to this citizen science project I have also co-authored a paper on Crowdsourcing for atmospheric science applications that is currently in press.

I work within the Air Quality Group at the University of Leicester and one of our current projects is the development of some open source air quality sensors. These will be designed to be cheap but also scientifically sound. A significant test period is underway with the current version of these sensors and so far they have been installed in electric cars and elsewhere in Leicester and Berlin (a sensor was installed in Berlin during OKFest). We are still finalising the designs of these but plan to release an open source design in 2015. When we have final designs for these I am planning to install several across schools in Leicester as they offer a much cheaper alternative to those sensors currently on the market.

What next?

This fellowship has been a great opportunity for me to kick start a citizen science project and further my interest in the development of open source air quality sensors. It also allowed me to attend conferences to discuss open data in air quality which received positive feedback from many colleagues.

The next steps for me are to continue with my school based project and aim this year to run a sensor in a school for a full term. I am also hoping to extend this to more schools. I also plan to continue to be involved with other citizen science projects where time will allow and continue to promote open data in air quality.

I have thoroughly enjoyed my year as a Panton Fellow and would like to finish by thanking Peter Murray-Rust, Michelle Brook, Jenny Molloy and my fellow fellows for all their support and ideas over the year.

Outputs:

Introduction for schools

Conference Poster presented at the Air Quality Conference 2014

General information poster for schools

Presentation from NCEO Young Scientists Conference

Blogs

http://science.okfn.org/2013/11/01/my-first-month-as-a-panton-fellow/

http://science.okfn.org/2013/12/11/citizen-science-project-for-air-quality-measurements/

http://science.okfn.org/2014/02/19/a-live-aq-data-feed-finally/

http://science.okfn.org/2014/06/10/recruiting-scientists/

Reports

http://science.okfn.org/2013/10/03/a-quick-hello-from-a-panton-fellow/

http://science.okfn.org/2014/01/08/an-update-on-my-panton-fellowship/

http://science.okfn.org/2014/04/11/panton-progress-at-the-half-way-point/

Papers

Illingworth, S.M, Muller, C.L, Graves, R and Chapman, L., UK Citizen Rainfall Network: a pilot study, Weather, 2014, 26:8, 203-207

Muller, C.L, Champman, L., Johnston, S., Kidd, C., Illingworth, S., Foody, G., Overeem, A., Graves, R., Crowdsourcing for Climate and Atmospheric Sciences: Current Status and Future Potential, IJOC, 2014, In Press

Podcasts

NERC Planet Earth Online podcast, http://planetearth.nerc.ac.uk/multimedia/story.aspx?id=1735&cookieConsent=A

Media

Electric Cars and air quality story

No Comments »

Panton Fellowship: End-of-Year Round-up

Samuel Moore - November 10, 2014 in Featured, Panton Fellowships

This is cross-posted from Samuel Moore’s blog, Scholarly Skywritings.

My time as a Panton Fellow has been both busy and extremely rewarding. In the last year I’ve been involved in a number of projects, met some fantastic people and attended a number of events centered on data sharing within academia. Whilst data sharing in the humanities and social sciences is still in a very nascent stage, especially the average researcher’s awareness of open data, there is definitely a sense that it is firmly on the agenda as part of the broader move towards openness in scholarly research. The crucial thing now is to continue to reach out to the average researcher, highlighting the benefits that open data offers and ensuring that there is a stock of accessible resources offering practical advice to researchers on how to share their data.

Issues in Open Research Data

With this in mind, in this final post I had originally wanted to be able to share the open-access book I’ve commissioned entitled Issues in Open Research Data, but alas it is still in production and will be published in November. Nevertheless, I am delighted to say that the book was successfully funded via the crowd-funding website Unglue.It and will be available in PDF, EPUB and low-cost print editions when it is published. The book features chapters by open data experts in a range of academic disciplines, covering practical information on licensing, ethics, and advice for data curators, alongside more theoretical issues surrounding the adoption of open data.

As the book will be open access, each chapter will be able to standalone from the main volume so that communities can host, distribute, build upon and remix the content. The book is primarily a work of advocacy and aims to start a conversation with the academic community at large – I’ll be sending out copies to research libraries, repositories and others that might be interested. Do get in touch if you think your institution would like a printed copy and I’ll see what I can do.

Journal of Open Humanities Data

Another initiative I wanted to mention is the forthcoming Journal of Open Humanities Data, which will be launching very soon through Ubiquity Press. The journal will feature peer-reviewed publications describing humanities data or techniques with high potential for reuse, everything from cultural items to large text corpora. In doing this, the journal aims to incentivise data sharing through publication credit, which in turn makes data citable through usual academic paper citation practices. Ultimately the journal will help researchers share their data, recommending repositories and best practices in the field, and will also help them track the impact of their data through citations and altmetrics. The call for papers will be posted in the next few weeks but, again, please do get in touch if you’d like to hear more.

Thanks!

Last of all, many thanks to the Open Knowledge Foundation for all their advice and support: specifically, Peter Murray-Rust, Michelle Brook, Jenny Molloy and Jonathan Grey, and many others too. I have already signed up to be involved in a few Open Knowledge projects in the coming year and I look forward to helping further the cause of openness across academia (and maybe working on my PhD..!)

Here is a roundup of some of the activities I’ve been involved in over the past year:

On stage at the Open Science Panel Vienna (Photo by FWF/APA-Fotoservice/Thomas Preiss)

It’s hard to believe that it has been over a year since Peter Murray-Rust announced the new Panton fellows at OKCon 2013. I am immensly proud that I was one of the 2013/14 Panton Fellows and the first non UK-based fellow. In this post, I will recap my activities during the last year and give an outlook of things to come after the end of the fellowship. At the end of the post, you can find all outputs of my fellowship at a glance. My fellowship had two focal points: the work on open and transparent altmetrics and the promotion of open science in Austria and beyond.

Open and transparent altmetrics

The blog post entitled “All metrics are wrong, but some are useful” sums up my views on (alt)metrics: I argue that no single number can determine the worth of an article, a journal, or a researcher. Instead, we have to find those numbers that give us a good picture of the many facets of these entities and put them into context. Openness and transparency are two necessary properties of such an (alt)metrics system, as this is the only sustainable way to uncover inherent biases and to detect attempts of gaming. In my comment to the NISO whitepaper on altmetrics standards, I therefore maintained that openness and transparency should be strongly considered for altmetrics standards.

In another post on “Open and transparent altmetrics for discovery”, I laid out that altmetrics have a largely untapped potential for visualizaton and discovery that goes beyond rankings of top papers and researchers. In order to help uncover this potential, I released the open source visualization Head Start that I developed as part of my PhD project. Head Start gives scholars an overview of a research field based on relational information derived from altmetrics. In two blog posts, “New version of open source visualization Head Start released” and “What’s new in Head Start?” I chronicled the development of a server component, the introduction of the timeline visualization created by Philipp Weißensteiner, and the integration of Head Start with Conference Navigator 3, a nifty conference scheduling system. With Chris Kittel and Fabian Dablander, I took first steps towards automatic visualizations of PLOS papers. Recently, Head Start also became part of the Open Knowledge Labs. In order to make the maps created with Head Start openly available to all, I will set up a server and website for the project in the months to come. The ultimate goal would be to have an environment where everybody can create their own maps based on open knowledge and share them with the world. If you are interested in contributing to the project, please get in touch with me, or have a look at the open feature requests.

Evolution of the UMAP conference visualized in Head Start. More information in Kraker, P., Weißensteiner, P., & Brusilovsky, P. (2014). Altmetrics-based Visualizations Depicting the Evolution of a Knowledge Domain 19th International Conference on Science and Technology Indicators (STI 2014), 330-333.

Promotion of open science and open data

Regarding the promotion of open science, I teamed up with Stefan Kasberger and Chris Kittel of openscienceasap.org and the Austrian chapter of Open Knowledge for a series of events that were intended to generate more awareness in the local community. In October 2013, I was a panelist at the openscienceASAP kick-off event at University of Graz entitled “The Changing Face of Science: Is Open Science the Future?”. In December, I helped organizing an OKFN Open Science Meetup in Vienna on altmetrics. I also gave an introductory talk on this occasion that got more than 1000 views on Slideshare. In February 2014, I was interviewed for the openscienceASAP podcast on my Panton Fellowship and the need for an inclusive approach to open science.

In June, Panton Fellowship mentors Peter Murray-Rust and Michelle Brook visited Vienna. The three-day visit, made possible by the Austrian Science Fund (FWF), kicked off with a lecture by Peter and Michelle at the FWF. On the next day, the two lead a well-attended workshop on content mining at the Institute of Science and Technology Austria.The visit ended with a hackday organized by openscienceASAP, and an OKFN-AT meetup on content mining. Finally, last month, I gave a talk on open data at the “Open Science Panel” on board of the MS Wissenschaft in Vienna.

I also became active in the Open Access Network Austria (OANA) of the Austrian Science Fund. Specifically, I am contributing to the working group “Involvment of researchers in open access”. There, I am responsible for a visibility concept for open access researchers. Throughout the year, I have also contributed to a monthly sum-up of open science activities in order to make these activities more visible within the local community. You can find the sum-ups (only available in German) on the openscienceASAP stream.

I also went to a lot of events outside Austria where I argued for more openness and transparency in science: OKCon 2013 in Geneva, SpotOn 2013 in London, and Science Online Together 2014 in Raleigh (NC). At the Open Knowledge Festival in Berlin, I was session facilitator for “Open Data and the Panton Principles for the Humanities. How do we go about that?”. The goal of this session is to devise a set of clear principles which describe what we mean by Open Data in the humanities, what these should contain and how to use them. In my role as an advocate for reproducibility I wrote a blog post on why reproducibility should become a quality criterion in science. The post sparked a lot of discussion, and was widely linked and tweeted.

by Martin Clavey

What’s next?

The Panton Fellowship was a unique opportunity for me to work on open science, to visit open knowledge events around the world, and to meet many new people who are passionate about the topic. Naturally, the end of the fellowship does not mark the end of my involvement with the open science community. In my new role as a scientific project developer for Science 2.0 and open science at Know-Center, I will continue to advocate openness and transparency. As part of my research on altmetrics-driven discovery, I will also pursue my open source work on the Head Start framework. With regards to outreach work, I am currently busy drafting a visibility concept for open access researchers in the Open Access Network Austria (OANA). Furthermore, I am involved in efforts to establish a German-speaking open science group

I had a great year, and I would like to thank everyone who got involved. Special thanks go to Peter Murray-Rust and Michelle Brook for administering the program and for their continued support. As always, if you are interested in helping out with one or the other project, please get in touch with me. If you have comments or questions, please leave them in the comments field below.

All outputs at a glance

Head Start – open source research overview visualization

Blog Posts

Audio and Video

Panton Fellows introduction at OKCon 2013
Panel “Science in a time of change – Is Open science the future?” [German]
Podcast Open Science in Research [German]
Introduction to Open Research Data as part of Open Science Panel Vienna [German]

Slides

Reports

Open Science Sum-Ups (contributions) [German]

The third quarter of my Panton Fellowship in the rear view mirror

Three quarters down in my Panton Fellowship, it is time again to review my activities.

The open source visualization Head Start, which gives scholars an overview of a research field, remained one of my focal points. In April, I released version 2.5 which includes a brand new server component that lets you manipulate the visualization after it has loaded. The new version also contains the timeline visualization created by Philipp Weißensteiner, along with a consolidated code base and many bug fixes. Furthermore, I worked on the integration of Head Start with Conference Navigator 3, a nifty scheduling system that allows you to create a personal conference schedule by bookmarking talks from the program. Head Start will be used as an alternate way of looking at the topics of the conference, and to give better context to the talks that you already selected and the talks that are recommended for you. Finally, in the wake of Peter Murray-Rust’s visit to Vienna in June (more on that later), I teamed up with Chris Kittel and Fabian Dablander to take first steps towards automatic visualizations of PLOS papers. The accompanying branch can be found here.

by opensource.com

I also continued to promote open and transparent altmetrics. In the blog post entitled “All metrics are wrong, but some are useful”, I argued that no single number can determine the worth of an article, a publication, or a researcher. Instead, we have to find those numbers that give us a good picture of the many facets of a paper and put them into context. In my comment to the otherwise excellent NISO whitepaper on altmetrics standards, I maintained that openness and transparency should be strongly considered for altmetrics standards. This is the only way to uncover biases inherent in all metrics. It would also make it easier to uncover attempts of gaming the system.

A highlight of the last quarter was Peter Murray-Rust’s and Michelle Brook’s visit to Vienna. The three-day visit, made possible by the Austrian Science Fund (FWF), kicked off with a lecture by Peter and Michelle at the FWF. A video of the great talk entitled Open Notebook Science can be found here. On the next day, the two lead a well-attended workshop on content mining workshop at the Institute of Science and Technology Austria.The visit ended with a hackday organized by openscienceASAP, and an OKFN-AT meetup on content mining, with presentations by PMR, Andreas Langegger (Zoomsquare), Roman Kern (Know-Center) and Marion Breitschopf (meineabgeordneten.at). It was a very enlighting yet intense week, as you can also read in PMR’s account of the activities.

Last but not least, I attended a meeting of the Open Access Network Austria working group on outreach. There, I will lead an effort to come up with a concept for enhanced visibility of open access efforts. Finally, I also contributed to the open science sum-ups of activites in Austria, Germany and beyond. Here you can find the monthly summaries for March, April and May [in German].

Sadly, the next quarter will also be the last of my Panton Fellowship, but a first highlight is already lurking around the corner: the Open Knowledge Festival will kick off in Berlin on July 15. See you all there!

Recruiting Scientists

Working out where we should install our sensors

Anyone whose been following the progress of my fellowship through my blog posts will know that I have been working towards getting sensors into schools for a while now. Well a couple of weeks ago I finally ran an introductory session with some primary school pupils (aged 8-11) at Kibworth CE Primary school in Leicestershire.

I had been developing the introductory material for a few weeks prior to the lesson with some help from the teachers at Kibworth who have been really responsive and open to my ideas. We decided we wanted this activity to be very student led so that they actually planned much of the experiment themselves to encourage them to think about why we were doing this in more depth. We titled the introductory session “What’s in the air you breathe?”.

Snapshots of the introductory presentation “What’s in the air you breathe”

I started the session by introducing the topic of air quality to the students, from the very basic first discussions of what makes up the air to talking about emission sources and health effects of air pollution. The introduction lasted less than 20 minutes and I encouraged lots of discussion with the students, asking them specific questions to work out what knowledge they had and to allow them to teach one another. The response to this was great and I was impressed by how much they knew about the atmosphere, one student explained the greenhouse effect to us and another mentioned the ozone hole. I hadn’t expected them to know so much about the topics we were discussing and so I was really pleased when I started talking to them.We then showed the students the equipment that they would have in school and explained what everything did. It was then over to the students to work out in groups where they wanted to install all of the sensors. To make this decision I asked them to think about where they thought the sources of air pollution around the school would be and where there are people who would be breathing it in. They quickly identified that the highest levels of pollution were likely to be in the car park, near the road and at the bottom of the playground which was relatively close to a train line. They also told me that in the morning and afternnon lots of people would be walking through the car park and at lunchtimes the students would all be in the playground. At this point one of the fundamental hurdles of being a field work scientist had to also be explained to the students- some of the sensors need mains power and so although the school gates may have been a good position in terms of producing interesting data, logistically it wasn’t possible to power the sensor that far from the school building.After lots of enthusiastic discussion and some expectation management they decided that they would like to put the sensor in three positions and so the pupils planned to move it around the school during the term. These were:

In the playground near to the car park and the chicken coop- they wanted to see what levels of pollution the chickens were being exposed to as well as themselves during playtime.
At the bottom of the playground near to the train tracks.
In the main playground where most of the students played at lunchtimes.

Lots of enthusiastic ideas…

The sensors are now with the school waiting to be installed in the next few weeks at which point data will start streaming in. While the students are busy being the scientists I need to get on with planning a data analysis session that we can run before the summer holidays. Overall I’m really pleased with how the session went and look forward to going back into the school soon.

All metrics are wrong, but some are useful

by Leo Reynolds

Altmetrics, web-based metrics for measuring research output, have recently received a lot of attention. Started only in 2010, altmetrics have become a phenomenon both in the scientific community and in the publishing world. This year alone, EBSCO acquired PLUM Analytics, Springer included Altmetric info into SpringerLink, and Scopus augmented articles with Mendeley readership statistics.

Altmetrics have a lot of potential. They are usually earlier available than citation-based metrics, allowing for an early evaluation of articles. With altmetrics, it also becomes possible to assess the many outcomes of research besides just the paper – meaning data, source code, presentations, blog posts etc.

One of the problems with the recent hype surrounding altmetrics, however, is that it leads some people to believe that altmetrics are somehow intrinsically better than citation-based metrics. They are, of course, not. In fact, if we just replace the impact factor with the some aggregate of altmetrics then we have gained nothing. Let me explain why.

The problem with metrics for evaluation

You might know this famous quote:

“All models are wrong, but some are useful” (George Box)

It refers to the fact that all models are a simplified view of the world. In order to be able to generalize phenomena, we must leave out some of the details. Thus, we can never explain a phenomenon in full with a model, but we might be able to explain the main characteristics of many phenomena that fall in the same category. The models that can do that are the useful ones.

Example of a scientific model, explaining atmospheric composition based on chemical process and transport processes. Source: Strategic Plan for the U.S. Climate Change Science Program (Image by Phillipe Rekacewicz)

The very same can be said about metrics – with the grave addition that metrics have a lot less explanatory power than a model. Metrics might tell you something about the world in a quantified way, but for the how and why we need models and theories. Matters become even worse when we are talking about metrics that are generated in the social world rather than the physical world. Humans are notoriously unreliable and it is hard to pinpoint the motives behind their actions. A paper may be cited for example to confirm or refute a result, or simply to acknowledge it. A paper may be tweeted to showcase good or to condemn bad research.

In addtion, all of these measures are susceptible to gaming. According to ImpactStory, an article with just 54 Mendeley readers is already in the 94-99 percentile (thanks to Juan Gorraiz for the example). Getting your paper in the top ranks is therefore easy. And even indicators like downloads or views that go into the hundreds of thousands can probably be easily gamed with a simple script deployed on a couple of university servers around the country. This makes the old citation cartel look pretty labor-intensive, doesn’t it?

Why we still need metrics and how we can better utilize them

Don’t get me wrong: I do not think that we can come by without metrics. Science is still growing exponentially, and therefore we cannot rely on qualitative evaluation alone. There are just too many papers published, too many applications for tenure track positions submitted and too many journals and conferences launched each day. In order to address the concerns raised above, however, we need to get away from a single number determining the worth of an article, a publication, or a researcher.

One way to do this would be a more sophisticated evaluation system that is based on many different metrics, and that gives context to these metrics. This would require that we work towards getting a better understanding of how and why measures are generated and how they relate to each other. In analogy to the models, we have to find those numbers that give us a good picture of the many facets of a paper – the useful ones.

As I have argued before, visualization would be a good way to represent the different dimensions of a paper and its context. Furthermore, the way the metrics are generated must be open and transparent to make gaming of the system more difficult, and to expose the biases that are inherent in humanly created data. Last, and probably most crucial, we, the researchers and the research evaluators must critically review the metrics that are served to us.

Altmetrics do not only give us new tools for evaluation, their introduction also presents us with the opportunity to revisit academic evaluation as such – let’s seize this opportunity!

Panton Fellow Update: Introduction to Open Research Data

In my first three-month update report report I discussed the book I’m working on as the major output of my Panton Fellowship. Entitled Introduction to Open Research Data, the book explores both the practical and theoretical issues associated with Open Data from a range of general and disciplinary viewpoints. The book will be Open Access, available in various ebook formats and low-cost print editions, and remixing will be encouraged – particularly the subject-specific guidance, which disciplinary communities can build upon as a foundation for a collection of resources on Open Data.

Whilst I am still awaiting a couple of contributions, I am happy to be able to share a provisional table of contents for the book. (Chapter topics on the left and authors on the right . Chapter titles still TBD):

1. Foreword: Introduction to the Panton Fellowships
2. Introduction to the book and the Panton Principles – Sam Moore (with input from the original Panton group)
3. Open Content Mining – Peter Murray-Rust and Jenny Molloy
4. Open Data and Neoliberalism – Eric Kansa
5. Data Sharing in a Humanitarian Organization: The Experience of Médecins Sans Frontières – Unni Karunakara (previous published in PLOS Medicine)
6. Open Data in Earth/Climate Sciences – Sarah Callaghan
7. Open Data in Psychology – Wouter van den Bos, Mirjam Jenny and Dirk Wulff
8. Digital Humanities and Linked Open Data – Jodi Schneider
9. Open Data in Palaeontology – Ross Mounce
10. Open Data in the Health Sciences – Tom Pollard
11. Open Data in Economics – Velichka Dimitrova
12. Why Open Drug Discovery Needs Four Simple Rules for Licensing Data and Models – Antony J. Williams, John Wilbanks and Sean Ekins (previously published in PLOS Computational Biology)

I won’t go into more detail about the content of each chapter, though authors were given free rein to approach the subject however they saw fit. Furthermore, I sought permission from the authors of the previously published pieces, though they were originally published under CC BY, and all were happy for their contributions to appear in the book.

I’m super excited for how this is coming together and I hope to have the book published by August. I will of course be posting updates along the way. Get in touch if you have any questions!

What’s new in Head Start?

The past couple of months I have been working on the open source visualization Head Start in the context of my research stay at University of Pittsburgh. Head Start is intended for scholars who want to get an overview of a research field. They could be young PhDs getting into a new field, or established scholars who venture into a neighboring field. The idea is that you can see the main areas and papers in a field at a glance without having to do weeks of searching and reading. You can find more information in my last blog post on the system.

If you read this post, you already know that Philipp Weißensteiner introduced a timeline visualization to the repository that lets you compare different datasets in a single view. I finished the integration of the timeline visualization, making it possible to review all datasets both in the regular and the timeline view. I was also busy consolidating the code and fixing the occassional bug along the way. The biggest change in version 2.5, however, is the introduction of a server component to Head Start. So far, Head Start consisted of a pre-processing system for generating the data, and an HTML5 interface for visualizing the data. There was no way of manipulating the visualization after it had been loaded. The new server component consists of REST-ful webservices and a PHP backend to deal with dynamic requests.

The server component proved very useful during the integration of Head Start with Conference Navigator 3, developed by the great folks of the PAWS Lab here in Pittsburgh. Conference Navigator is a nifty scheduling system that allows you to create a personal conference schedule by bookmarking talks from the program. The system then gives you recommendations for further talks based on your choices. Head Start will be used as an alternate way of looking at the topics of the conference, and to give better context to the talks that you already selected and the talks that are recommended for you. To do that, Head Start dynamically loads bookmarking and recommendation data from the CN3 database.

What’s next? First of all, the system will be evaluated with users in one of the upcoming conferences that Conference Navigator supports. Furthermore, I would like to move the preprocessing systems from an offline to an online solution, thus enabling it to load live content from APIs.

If any of the above got you interested, here is the link to the Github repository. As always, please get in touch if you have any questions or comments, or in case you want to collaborate on this project.

Public Health Data: as Open as it can be?

I recently sent out invitations for a forthcoming article collection entitled Exemplar Public Health Datasets, to be published in the recently launched journal Open Health Data. The collection will feature peer-reviewed articles describing public health datasets as part of the Enhancing discoverability of public health and epidemiology research data project. Funded by the Wellcome Trust, the project seeks to appraise the ways in which public health datasets could be made easier for potential users to discover, and this article collection is one way of exploring the issue.

The collection will be composed of Data Papers, which are publications designed to make other researchers aware of data that is of potential use to them. Importantly, a data paper does not replace a research article, but rather complements it. As such, the data paper describes the methods used to create the dataset, its structure, its reuse potential, and a link to its location in a repository.

However, one issue that immediately presented itself is that most public health research data is not collected in a way that allows open sharing. Public health research often takes the form of large-scale longitudinal studies involving numerous research groups, during which a great deal of patient data is collected. Whilst the data are anonymised, there are always concerns surrounding de-identification, especially given the sensitive nature of the material, and so data is shared only to those who meet the accessibility criteria. As Jones et al. write, regarding the Secure Anonymous Information Linkage (SAIL) Gateway:

‘Even though the data are anonymised, someone with legitimate access to the data, or a potential intruder, may attempt to re-identify individuals or clinicians. It is essential, therefore, that anonymisation is robust, that measures to further encrypt key variables are in place, and that data presented can be limited to the needs of a given project.’ [1]

Because of this, data sharing in public health is approached with extreme caution and there are many disincentives for doing so. The Exemplar Public Health Datasets collection aims to change this by formalising the process for data access. For example, if there are accessibility criteria associated with a particular dataset, a Data Paper would be a great place for outlining the criteria, the location of the dataset, and steps needed to access it. What’s more, whilst the data itself might not be shareable, there is still a great deal of value in openly sharing consent forms, metadata and related protocols. The Data Paper format encourages the sharing of all elements related to the research lifecycle, aiming to reach a position where ‘Open’ is the default for public health research whilst still negotiating the complex world of access to patient data.

Get in touch if you have any questions!

[1] Jones et al. ‘A case study of the Secure Anonymous Information Linkage (SAIL) Gateway: A privacy-protecting remote access system for health-related research and evaluation’ Journal of Biomedical Informatics (in press) http://doi.org/10.1016/j.jbi.2014.01.003