text:
# Section 4 Data Management Workflow

This section is in two parts, to first describe a workflow that we use internally at TBEP to manage our data in section 4.1 and then to describe a road map for opening internal or external datasets at your own organization in section 4.2. The first section expands on our philosophy for using open science to manage data, including specific workflows we use, as context to the second section. Our approach is one way of applying open science to managing data. Applying the same approach at your organization may or may not be appropriate depending on your internal and external needs for managing data. As such, our approach is generalizable and modular - any of the approaches can be modified in part or together for your own needs.

## 4.1 The TBEP approach

### 4.1.1 An Open Science philosophy

Sections 2.3 and 2.4 introduced you to our basic philosophy and approach to managing data at TBEP. As an organization that facilitates science, management, and outreach activities among our local partners, we adopt open science as a cornerstore strategy that will serve the Program’s core values. This approach is made explicit in our Strategic Plan that describes how we achieve programmatic goals defined under our Comprehensive Conservation and Management Plan (CCMP) and the partner commitments identified in our Interlocal Agreement to help us achieve our goals.

Our data Quality Management Plan (QMP, Sherwood et al. (2020)) is a companion document to this SOP that ensures the data used by TBEP for decision-making has known and documented quality and is being used appropriately. The QMP establishes an internal process for verifying data quality standards that conform with the federal requirements our organization is subject to, as funded in part by federal dollars under Section 320 of the Clean Water Act. Conversely, this SOP is a more hands-on and accessible document that describes a how-to approach for data management that we adopt as an organization. The SOP goes beyond the QMP by exposing the process and ideas behind how we manage data at TBEP so that others can learn from our experience. We encourage you to also view our QMP to understand the literal benchmark we use to ensure quality of our data.

We actively work to apply open science to every activity we pursue to achieve our goals under the CCMP. Open science is a philosophy and set of tools to make research reproducible and transparent, in addition to having long-term value through effective data preservation, provenance and sharing (Beck et al. 2020). We use a definition from the Creative Commons for open science as:

Practicing science in such a way that others can collaborate and contribute, where research data, lab notes and other research processes are freely available, under terms that enable reuse, redistribution and reproduction of the research and its underlying data and methods.

There are several key words we can extract from the definition - collaborate, contribute, reuse, redistribute, and reproduce. These concepts channel some of the ideas described by the FAIR principles (section 3.2). We can further elaborate on these key words by defining open science as a set of four core principles (Dan Gezelter, http://openscience.org/what-exactly-is-open-science/.

1. Transparency in experimental methods, observations, and collection of data.
2. Public availability and reusability of scientific data.
3. Public accessibility and transparency of scientific communication.
4. The use of web-based tools to facilitate scientific collaboration and reproducibility.

Why is this so important? Environmental science is very much in the business of applied science, meaning that research that is conducted to develop an understanding of the environment can be used to support the protection and management of a resource. We need to understand a problem before we can pursue actions that effectively adress it, especially if a wrong decision or action is costly. Active and useful channels of communication must exist for the lessons learned from science to be applied to real world problems. Applied science can be facilitated with open science to create these channels, and therby accelerate practical solutions.

Without getting too much into the history of how insular practices among academics have contributed to closed science, it’s useful to briefly discuss some of reasons why science may not be translated into action. As a generalization, researchers are trained to study and document details. Progress in science is based on: 1) an intimate understanding of details that guide process, and 2) convincing your peers through rigorous review that you actually understand the details you claim to understand. As a result, we catalog progress in ways that are true to the scientific process, often as dense texts with every last detail noted. Many researchers not being taught otherwise will often assume that this is an effective way to communicate scientific results to non-scientists. What we don’t realize is that those that need this information to make decisions do not communicate this way because they are not in the business of scientific discovery. Unless they have a personal interest, they don’t care about the science behind the decision, only that the science is right to justify the decision. The most ineffective approach for a scientist to inform environmental management is to deliver a dense 500 page report and assume it provides an effective vehicle for an environmental manager to make a rational decision. This is not applied science - it is “implied science” because we implicitly decide that our conventional modes of scientific communication will influence management or policy.

In addition to communication barriers, other challenges to applied science include irreproducible results, information loss, inaccessible data, and opaque workflows (section 2.2, Figure 4.1). These challenges affect how science is delivered to decision-makers, how much trust a decision-maker can have in the science behind the decision, and how likely the science can be used as a springboard for additional scientific inquiry. Effective data management as a subset of the broader principles of open science can help bridge the “research-management divide” and help develop continuity of scientific products that can benefit the larger research community.

### 4.1.2 The open science cake

Truly applied science facilitated by open science principles allows for research results or data to connect with different audiences along a spectrum. It allows research to be shared with other researchers, be connected with decision-makers, and be accessible to the general public. Where an individual consumes scientific information along the spectrum depends on their interest, need, or level of background knowledge about a subject. A solid technical foundation is a prerequisite for sharing information and open science methods allow various elements of the research foundation to be accessible to different end users. We meet our audience where they’re at, rather than assuming they can find their way to the details they need.

We can describe this metaphor as the open science cake (figure 4.2). We use this metaphor because everybody loves cake and it conveniently describes the TBEP philosophy to delivering science in an applied context. This delicious layered cake is a gradient of information from top to bottom. At the top, the information is more general (e.g., educational material for public consumption) or can be used to inform action (e.g., what needs to be done to remedy a problem). At the bottom, the information has specificity and forms the foundation for generality or action at the upper layers. The bottom of the cake is large, reflecting the decades of research and technical resources that are available to inform the management of Tampa Bay (e.g., our library). The bottom also includes resources that can be used to springboard additional research, such as analysis code and source datasets. Individuals at the top of the cake probably don’t want a slice at the bottom, but the slice they take from the top would not exist without support from the bottom.

Most of our partners are professionals from resource management or local government agencies that have some vested interest in the protection and restoration of Tampa Bay. This is the primary audience that we can inform for decision-making. Broadly speaking, this is the audience that needs distilled information from research products but with a level of specificity that goes beyond educational materials. These individuals are in the middle of the cake and the slices they take are actionable science products, such as interactive dashboards, automated report cards, and other decision support tools. The middle part of the cake is where conventional science becomes truly applied science.

The cake also emphasizes a vertical connection among the layers that allows an individual to take a slice as high or as low in the cake as they want. This is a critical principle of open science that speaks to accessibility of information at all levels of the scientific process. Most of the time, an individual will take a slice from the cake at the level that’s appropriate for their needs. However, we want our science (and data) to be transparent and accessible under the FAIR principles and someone can take a slice at a different level, if they have a need to do so. This also speaks to developing a community of practice for open science - we develop this community to provide easier access to the tools at the bottom of the cake and develop the ability to use them to reproduce or expand on existing products at all levels.

Our web products on the data visualization section of our web page are designed to guide an individual to the slices they need at the different levels of the cake. The website is setup as a series of cards (cakes) for each reporting product that act as an entryway (top of the cake) to the middle and bottom layers of each cake. For example, if someone clicks on the Water Quality Report Card, they are directed to a web page that has very general information about the reporting product and links to a summary pdf that distills over forty years of water quality data for the Bay. There are links on the right side of the page that provide access to the building blocks of the report card, including the online dashboard, source code for the report card, build status of the report (more on this in section 4.1.3), citable DOI, and technical documents that describe the science behind our water quality assessment approach. These links provide the path to the lower levels of the cake.

### 4.1.3 How do we build the cake?

The cake is a useful metaphor to describe how we apply open science to achieve applied science, but how is this done in practice? How are the layers of the cake actually linked to one another? We use several open source programming tools to link source data to reporting products, with the goal of producing the most timely information for decision-makers with minimal overhead by internal staff. In this section, we describe these tools and how we link them together to create a workflow that is both automated and reproducible.

The workflow we use to link source data to reporting products for our annual water quality assessment is shown in figure 4.3. The process begins by accessing an external data source from our partners. In this case, this workflow accesses a spreadsheet of water quality data on an FTP site maintained by the Environmental Protection Commission of Hillsborough County (EPC). These data are processed using custom-built software developed by TBEP staff and with a suite of more general open source tools, including R, RStudio, relevant data wrangling packages, and tools for document preparation. The open source tools we’ve created are also hosted online on GitHub which serves two goals. First, providing the tools on GitHub makes them discoverable and accessible to others. Second, they are integrated into an automated process to make sure the most current data are used for reporting. Once the build process for the report card is done, the final products are shared as a Shiny web application for additional inquiry and discovery, and our two-page PDF report are hosted on the TBEP website for general dissemination.

The core component of this entire workflow is the tbeptools R package (Beck et al. 2021). This software was developed using the R programming language to read, analyze, and visualize data that we use to report on progress in achieving CCMP goals. Most of the data tools on our web page depend on functions within the tbeptools package to work with the raw data provided from our partners. Although tbeptools is primarily used by TBEP staff, the package is provided free of use (under the MIT license) for anyone interested in exploring the data on their own. Importantly, all source code is available on GitHub so that anyone with an interest can understand exactly what is done to process the data we use for reporting. This is a very literal definition of method transparency.

There are several functions in the tbeptools package that are built specifically for reporting on water quality, all of which are explained in detail in the introduction vignette for the tbeptools package. A “vignette” in the R world is a plain language document that explains how to use functions in a package. Currently, the tbeptools package includes five vignettes, one for each indicator that has reporting functions available in the package:

• Intro to TBEP tools: A general overview of the package with specific examples of functions for working with the water quality report card.
• Tampa Bay Nekton Index: An overview of functions to import, analyze, and plot results for the Tampa Bay Nekton Index.
• Tampa Bay Benthic Index: An overview of functions to import, analyze, and plot results for the Tampa Bay Benthic Index.
• Tidal Creeks Assessment: An overview of functions to import, analyze, and plot results for the assessment of tidal creeks in southwest Florida.
• Seagrass Transect Data: An overview of functions to import, analyze, and plot results for the seagrass transect data collected in Tampa Bay.

Each vignette is set up similarly by explaining the functions used to read, analyze, and visualize the data. In fact, every function name in the package is named with an appropriate prefix for what it does, e.g., read_transect() reads seagrass transect data, anlz_transectave() analyzes annual averages of seagrass frequency occurrence, and show_transect() shows a plot of the transect data. The examples in the vignette further explain how to use the functions and what each function does when working with the data.

The functions in tbeptools used to read data into R were all built to ensure the most recent data are used for analysis. Each data import function follows a decision tree shown in figure 4.4, where a set of internal checks are used to see if the data are available on your computer, compare the data to the online source, and download the most recent version if a local file doesn’t exist or your current file is out of date. This process also ensures that any downstream reporting products are using the most current data. For example, the web page for water quality assessment has a provisional report card that is based on the most recent water quality data available from EPC. Although the “official” report card is published at the beginning of each year, provisional data throughout the year can be used to assess water quality changes in near real time.

The workflows we’ve created that access source data to create reporting products depend on data being online in a stable location. This underlies the importance of proper data management practices. We cannot create and use the reporting products without a findable and accessible location for the source data. The data we use for our various indicators are distributed at different locations depending on who maintains the information. This includes a mix of FTP sites, Microsoft Access databases, JSON files, or geospatial data hosted through third party websites. The various locations, data formats, and depths of available metadata are a potential concern for long-term viability of these workflows. A majority of the locations where these data are found are not formal data archives or repositories and there are not any “official” standards for how these data are made available. Because of this, a long-term goal for TBEP and our partners is to work towards a shared data management infrastructure that more closely follows the FAIR principles.

A critical part of the workflow in figure 4.3 is automation. We have developed the functions in the tbeptools package with this in mind, i.e., making sure the most up to date data are used without having to manually download the data. We also leverage continuous integration/continuous deployment (CI/CD) services through GitHub that automate our workflows. CI/CD simply means that we’ve setup automated checks and processes based on different triggers that happen when we do something to a project that we’ve put on GitHub. For example, every time we push (upload) a change that we’ve made on a local version (i.e., on a personal computer) of tbeptools to the main repository on GitHub, a set of automated checks are used to make sure we didn’t break anything in the package. The “badges” you see on GitHub and in the data visualization section of the TBEP website indicate if the checks were successful or not. These badges are small indicator icons that are shown as green (checks passing), red (checks failing), or grey (no information). They are often included in GitHub repositories and can also be linked on other websites. These provide a quick scan of whether or not you should expect the package to work as intended if you download it from its current state in the repository.

Many of the README files for the different GitHub projects we maintain include one or more badges that indicate if our data products are working as we hope. For example, the README file for our tbeptools R package includes three badges that indicate the status of different CI/CD processes (figure 4.5). The first badge, R-CMD-Check, shows if the set of standard checks for R packages are passing. There are dozens of checks for R packages, including things like making sure the documentation is up-to-date, file sizes aren’t excessive, the file structure is setup correctly, the examples run without errors, etc. (see chapter 19 in Wickham and Bryan (2015) for a full description of these checks). The second “pkgdown” badge shows if the website for the package (built with the R pkgdown package; Wickham and Hesselberth (2020)) has been successfully built with the last change to the repository, i.e., does information on the website reflect the package in its current state? Finally, the “codecov” badge provides a general estimate of how much code in the package includes unit tests as part of best practices for software development. In this case, nearly all of the code (92%) is tested in one form or another.

The CI/CD workflows are completely customizable to suit the needs of a given project. The badges in the previous example simply indicate if the CI/CD checks for a package and its website are working correctly. These are often included in README files to give users peace of mind that our development processes are following accepted standards in the broader open source community. Other badges can indicate if a custom workflow is up to date, such as for our automated reporting products. The water quality report card has a provisional draft that uses the most recent dataset from EPC. The CI/CD process is setup to rebuild the pdf by running a custom “build” file that imports the data, analyzes the results, and creates the plots, all using function from tbeptools. The output graphics are embedded in a type of document preparation system that mixes plain text and code to dynamically generate a static pdf. All of this is accomplished in the build file, which is triggered daily through the CI/CD services on GitHub. The CI/CD badge for this repository indicates if the daily build was run and if the provisional pdf was successfully created. Many of our reporting products leverage these services and you can view the status of each product from the badges we’ve placed on our main TBEP website or on the README file for specific GitHub repositories.

### 4.1.4 More on Git and GitHub

GitHub is a foundational tool that is central to our data management workflow. We’ve described how it can be used as an intermediate solution for hosting data (section 3.5) and how we use it to share and automate our reporting workflows (section 4.1.3). Our use of GitHub aligns with our broader philosophy of using open science and here we explain some more general concepts about what GitHub can provide to our community to emphasize the value it can have for data management.

Many people describe Git and GitHub synonymously, but we need to distinguish between the two to develop an understanding of the different services each provides. First, Git is a formal version control software, whereas GitHub is an online platform for sharing code that uses Git. It’s possible to use Git without using GitHub (i.e., using version control only on your personal computer) and it’s possible to use GitHub without using Git (e.g, using GitHub to share a file). Naturally, using both Git and GitHub together can help leverage the benefits of each. The relationship between the two is very similar to that of R and RStudio. Using R by itself is okay, but the value to yourself and others of using RStudio as a vehicle for R will be greatly enhanced.

Version control is a way to track the development history of a project. It serves the joint purposes of: 1) formally documenting the changes that have been made to code or software, and 2) making sure that the development history is permanent. Documenting changes provides a transparent record for yourself and others, and establishing permanency ensures that any of the changes that are made can be vetted and accessed as needed. Using Git is extra work, but when you need it you’ll be glad you’ve invested your time wisely. Think of an instance where you’ve saved different versions of a file with different names because you don’t want to delete any of your old work. You end up with many extra files and no clear way to understand the origins or connections between the files. Git takes care of this for you by providing a navigable insurance plan for your project.

GitHub lets you share your files or code under Git version control in an online environment so that you and your collaborators can more easily work together. You can host multiple projects under version control, view the entire history of each project, and allow others to more easily work asynchronously and together on the same project. GitHub also has tools for tracking “issues” associated with different projects, providing a simple way to document questions, bug fixes, or enhancements. GitHub is a near perfect example of open source in practice. Anyone can view and potentially contribute to other people’s projects using a platform that ensures everything is fully documented and never erased.

GitHub also includes a variety of other tools that facilitate openness:

• Release tagging to assign formal version numbers to code, data or software.
• GitHub actions to create your own CI/CD workflows (our examples in section 4.1.3 use these tools).
• Integration with Zenodo for DOI assignments to give your project a stable and permanent address. You can see these links on many of our projects on GitHub (e.g., our water quality dashboard).
• Website hosting, as for our tbeptools R package.

All of this may sound very specific to software development, but GitHub can take you a long way towards adopting FAIR principles through better data management practices. The concepts that apply to version control for code and software have parallels for data management and many of the features to facilitate openness in GitHub can also apply to data. Making your data accessible, documenting the changes you’ve made over time, and establishing a permanent home (e.g., through Zenodo) can all be done with GitHub. We elaborate on a case study example using GitHub for data management in section 5.3.

GitHub also lowers the barrier to inclusion for engaging others in a project. Unless you work with a dedicated team of software or web developers, it’s very rare that your colleagues will have experience with Git or even know what it is (although this may be less common in the future). This doesn’t mean that others are excluded from contributing. For example, anyone can post issues for a project through the simple web interface provided by GitHub. Changes to source documents can also be made online that can be tracked through version control without having to use Git on your own (e.g., see our contributing guidelines for this document in section 1.1).

The TBEP has a group GitHub page where all of our projects exist, including the source content for this SOP. We do this for all of the reasons mentioned above and as an attempt to serve as an example of how open sharing can lead to better science in less time (Lowndes et al. 2017). Anyone can view our pages to understand the source code, see the changes we’ve made over time, and post issues/edit content to directly contribute. This has immense value for how we collectively work as a team and with our partners outside of TBEP.

## 4.2 How can you manage data?

This section is written as a road map for managing a data product, keeping in mind the list of tools and resources in section 4.1 that can be used along the way to develop the product. These tools can help you at different stages of the data management process to help build the layers of the open science cake. The guidance provided by Goodman et al. (2014) and Michener (2015) are also excellent resources presented as “simple rules” for working with data. Goodman et al. (2014) develops a metaphor of data as a living entity by describing rules for the care and feeding of scientific data. Michener (2015) focuses on rules for developing a data management plan. Many of the concepts and tools presented herein are elaborated upon in these two resources.

A road map to developing a plan and set of tools for delivering a data product is shown in figure 4.6. This map is presented as a hypothetical one-year project from beginning to end, but it can be applied to a project of any duration. The steps are separated along a general timeline with a notable distinction between steps occurring before and after data collection. It’s also very important to realize that “data collection” can have a literal interpretation as collecting data in the field or during an experiment, whereas collection can also be considered generically as the process of creating less conventional data products (e.g., workflows, modelling output/information, decision-support tools, section 3.1).

There are five general steps in the road map. Where you’re at in the project determines what conversation you’re having with yourself and your research team about managing the data product. At the beginning of a project in Step 1, goals and objectives are defined, potential data contributions are identified, and metadata concepts are introduced. Step 2 is developing the data framework, meaning you will identify how your data are structured within the goals of the project. Here you are thinking about data dictionaries or workflows you will use to generate the data product. In Step 3, you are actively collecting and curating data in a way that conforms to the framework you’ve developed. This step is iterative because it is where most of the work happens and you may need to rethink some of the ideas around data curation based on how the collection process works in reality. Steps 4 and 5 focus on identifying where the data are going to be maintained/stored and finally delivering the data at that location.

The road map in figure 4.6 can be used in parallel with other project timelines, such as those included in a scope of work. It is a separate but complementary approach that focuses specifically on data, as compared to other details associated with a research project (e.g., budget, field work, etc.). The road map also emphasizes that researchers should be proactive in thinking about their data deliverables, particularly regarding important contributions, appropriate formats, metadata concepts, and conformity with this TBEP SOP. The earlier these conversations happen in a project, the easier it will be to deliver a well-documented data product.

A researcher or research team will benefit most by using this road map from the inception of a project, although we realize that this may not be the norm and data curation is routinely considered an afterthought. For this reason, we present the following sections as a guide to using this road map at any stage of the research process.

### 4.2.1 I’m at the beginning of my project

In an ideal scenario, you are actively thinking about a plan for delivering your data product at the beginning of a research project. Data are the foundation of the project and discussing how you will work with it at the beginning will ensure that the foundation is on solid ground moving forward. Starting the conversation early can also normalize ideas about the importance of paying close attention to data curation. Data are often poorly managed because the value of proper data management to the individual or an organization may not be apparent. These values are also not immediately obvious unless you’ve spent a lot of time working with other people’s data. This includes conversations about the value of metadata. Discussing these details at the beginning of a project will establish a culture of stewardship that is parallel to the larger research process.

Appropriate questions to answer at the beginning and early phases of a project (steps 1 and 2, figure 4.6) can include the following.

What type of project am I working on and what products can I expect?

The answer to these questions could be based on the intended audience for results that are produced by this research. If your project will have results that can be applied to address some real-world issue, than you are likely working on a project that can deliver a data product to inform decision-making. Think about an appropriate format for this data product that will best meet the needs of your audience.

Which datasets are important?

The answers to the previous questions define your answer to this question. However, it’s a rare scenario when you only have one data product as the primary contribution from a project. Early conversations about which data products will be the focus of curation are necessary at this stage. Guidance for determining which datasets are important are expressed in detail in section 3.1.

How do I want to make the data accessible?

Data typically won’t be made accessible until the end of a project, but knowing where your data will live can help identify what tools you need to use along the way to deliver the product. This includes identifying an appropriate metadata format, analysis platforms to work with the data, and formats that are supported by the location where you’re going to keep the data. Having clear answers to these questions will save you the most time at the end of a project.

What QA protocols should be established?

Developing a plan for using the FAIR principles with your data is just as important as ensuring the data have adequate quality for use by you and others. At the beginning of a project, start actively developing a data management plan that defines appropriate methods for collecting the data and verifying its quality. Guidelines in Michener (2015) can help with this process or you can use the online Data Management Plan tool.

### 4.2.2 I’m somewhere in the middle of my project

In a less ideal scenario, you are somewhere in the middle of your project and are just now thinking about types of data products and the path you’ll take for their delivery at the end of the project. At this stage, you’ll need to retrofit some answers to the questions in the last section to identify important data contributions, how you’ll make them accessible, and how you’ll collect and format the data to ensure adequate quality. It will be easier to answer these questions if you have yet to collect any data. However, you will need to make sure that you and your research team understand the value of data curation since now it will be seen more as an afterthought that was not discussed at the beginning of the project. You will need to put in some extra work to normalize conversations around data management and why doing so is important.

If you have already collected data, you’ll need to take some time to evaluate how you’re collecting this information so that it addresses the answers to questions in section 4.2.1. If you’re collecting tabular data, make sure you are using a tidy data format (section 3.3) and that you have developed an appropriate data dictionary (section 3.4.3) to put some boundaries on how the information is stored. At this stage, it may be impractical and unreasonable to ask field or lab staff to enter their data differently and developing a post hoc workflow to wrangle your data into a tidy format may be the best option. Identifying appropriate keys for your data is also critical at this stage. If your data product is more general (e.g., analysis workflow, decision-support tool), make sure you can trace the provenance of these products to their conception. What inputs did you use to create this resource? If you built it from scratch, what tools are you using to build the product (e.g., software and versions)? You may need to do a bit of detective work to identify answers to these questions.

For metadata, the same rules apply in the middle of a project as those at the beginning of a project. First, identify where you plan on delivering and hosting your data and work backwards from there. If the location has a specific metadata standard, identify the information you’ll need to conform to that standard. If you plan on hosting your data at a more general location, start collecting answers to the “who”, “what”, “when”, “where”, and “how” questions for your data. The most important part of this process is understanding that metadata documentation should start as soon as you realize that you have not been doing so. The longer you wait, the more likely it is that you’ll be unable to track down information or you’ll simply forget important details about a dataset.

### 4.2.3 I’m at the end of my project

Prepare yourself for damage control if you’re at the end of your project and have yet to think about data management. You not only have to make some judgment calls on the most important contribution, but you also have to begin the laborious process of finding and extracting intimate details about that contribution. The temptation to identify low-hanging fruit that are easy to document but may not be the most important contribution is very real and you should avoid doing this to not undersell the potential impact of a project. Putting in the time to backlog the provenance of a more important contribution should be an investment that will have more pay off in the future than documenting an intermediate or less relevant data product. The most important question to answer at this stage is what is the most anticipated and impactful result of this research project and what dataset delivers this result?

In the real world, most researchers discuss delivery of a data product at the end of a project because nobody wants to do this at the beginning when more interesting and creative problems about a project are the center of discussion. Of course this is not ideal, but we want to encourage you not to feel overwhelmed and not to give up at this stage. We presented a lot of information about data management so far, but you should not feel that you need to do it all. As before, something is absolutely better than nothing and giving up because you are overwhelmed to do “all the things” should not be a deterrent. Open science and data management is incremental and, in reality, very few individuals will be able to complete all of the checklists without years of experience and substantial help from others. So, pick one thing off the list and use that as a starting point to building your comfort and skills in adopting better data management practices (spoiler: it should be metadata).

The following can be used as a generalized workflow for metadata generation, ideally at the beginning of a project. How far you go in these steps depends on where you want to store your metadata.

1. Identify which dataset(s) are important contributions of a project that you intend on sharing and that need metadata (see section 3.1).
2. Draft a general document to answer the “who”, “what”, “when”, “where”, “why”, and “how” questions in section 3.4.1 for each data product. This can be a simple text file that “accompanies” each data product. A spreadsheet entry form can also be useful so that the metadata are in tabular format. If it’s tidy, this can be used to import into a program for converting your metadata into a formal standard or for uploading to a data repository.
3. Convert the metadata document into an appropriate file format based on the metadata standard you’re using. For EML metadata, this would be an XML file.

This workflow is a starting point for creating simple to more complex metadata. For a hypothetical example, the absolute bare minimum for metadata might look like this (e.g., in a spreadsheet):

Who Marcus Beck ()
What Water Quality Assessment Data
When 2020
Where Tampa Bay
Why Reporing on annual water quality trends for programmatic update
How Data were collected monthly as part of the EPCHC monitoring program

If we wanted to add some specificity, we could create separate fields to include more detailed information.

Attribute Entry
first Marcus
last Beck
email
title Water Quality Assessment Data
startdate 1/1/2020
enddate 12/31/2020
location Tampa Bay, FL, USA
west -82.8
east -82.3
north 28
south 27.4
methods All data were from the Environmental Protection Commission of Hillsborough County (https://www.epchc.org/i-want-to/view-gis-maps-and-data). Data are monthly at mid-depth for each of 50 stations throughout the Bay from 1974 to 2020. Stations were visited on a rotating schedule such that one third of all stations were sampled each week. Bay segments represent management units of interest with distinct chemical and physical differences. Accordingly, station data were aggregated by median values within each of four Bay segments. Data are used to assess progress in achieving programmatic goals for the Tampa Bay Estuary Program.

Finally, if we wanted to convert this information to an EML metadata file, we can use some tools in the EML R package. This example includes all of the information from the last example, but using specific tags and entry methods for the EML format, The methods information can also be entered as a separate file for more long-form documentation (i.e., set_methods('data/methods.docx')). After the file is written with write_eml(), it can be validated for accuracy and completeness with eml_validate(). The finished metadata file can be viewed here.

library(EML)

me <- list(
individualName = list(givenName = 'Marcus', surName = 'Beck'),
)
title <- 'Water Quality Assessment Data'
coverage <- set_coverage(
begin = '2020-01-01',
end = '2020-12-31',
geographicDescription = 'Tampa Bay, Florida, USA',
west = -82.8, east = -82.3,
north = 28, south = 27.4)
methods <- set_methods('data/methods.docx')

my_eml <- eml$eml( packageId = uuid::UUIDgenerate(), system = "uuid", dataset = eml$dataset(
title = title,
creator = me,
contact = me,
coverage = coverage,
methods = methods
)
)

# write and validate the file
write_eml(my_eml, 'data/my_eml.xml')
eml_validate('data/my_eml.xml')

There are many more attributes that can be included in EML metadata. For example, we discussed the importance of a data dictionary in section 3.4.3. This information can be documented in an EML file in different ways and we encourage you to view the EML website for a full overview.

The intent of presenting this example was to demonstrate simple to more complex approaches that you can use to create metadata. However, it’s also worth pointing out that this process can be completely operationalized through the workflows described in section 4.1.3. It would be entirely conceivable to use a spreadsheet or web form entry to collect attributes for the EML file, import that information in R, and create the file with the EML R package. This can be automated through a CI/CD build process on GitHub, where a user would only have to enter data on a web form and the rest can be handled through a routine created in R that is executed on GitHub. Moving towards this approach to document and create metadata would be a tremendous leap forward in curating data products that are used by TBEP and its partners.

### 4.2.5 Let’s get it online!

The previous sections on sharing data have covered general topics on where data could live (section 3.5) and some options of how we do this at TBEP (section 4.1.4). These motivating examples are provided as a set of options, from simple to more complex, depending on how detailed and permanent you want to make your data. Your data are less likely to be orphaned when more time is invested in documenting metadata and if you choose a location that is legitimately setup for archiving (e.g., DataOne). The options we presented can be used to achieve these goals with varying success and it’s your perogative on which option is most appropriate given the time you have to dedicate to data management, the objectives you have in doing so, and the ultimate requirements specified in a particular project agreement.

We have not provided step-by-step details on how to deliver a data product online and we leave this exercise up to you. In most cases, getting a dataset online is straightforward but the ease of doing so depends on where you want to put the data. For simple solutions, such as FTP hosting or putting a dataset on Google Drive, all you need to do is upload the data by hand or use an existing file transfer service (e.g., PuTTy). Of course, make sure you also upload any metadata and make it clear that your metadata accompanies the file you uploaded. Hosting data on GitHub can be done using a command line interface (such as Git Bash) or as a GUI through GitHub Desktop. Tools available in RStudio can also be used to host projects on GitHub.

For more involved hosting platforms, data can also be uploaded through an online web interface. For example, data can be uploaded by hand to the knb node of DataOne in a web browser, including entry of metadata to generate the EML file. However, open source tools can also be used to link directly to these services to build an integrated data delivery pipeline to avoid uploading information manually. In addition to the EML R package, the metajam package can access repositories on DataOne using R. Many of these packages are under active development and the services they provide to access federated data repositories is expected to improve over time.

### References

Beck, M. W., C. O’Hara nad J. S. S. Lowndes, R. D. Mazor, S. Theroux, D. J. Gillett, B. Lane, and G. Gearheart. 2020. “The Importance of Open Science for Biological Assessment of Aquatic Environments.” PeerJ 8: e9539. https://doi.org/https://doi.org/10.7717/peerj.9539.

Beck, M. W., M. Schrandt, M. Wessel, E. Sherwood, G. Raulerson, and B. Best. 2021. Tbeptools: Data and Indicators for the Tampa Bay Estuary Program. https://github.com/tbep-tech/tbeptools.

Goodman, A., A. Pepe, A. W. Blocker, C. L. Borgman, K. Cranmer, M. Crosas, R. Di Stefano, et al. 2014. “Ten Simple Rules for the Care and Feeding of Scientific Data.” PLoS Computational Biology 10 (4): e1003542. https://doi.org/10.1371/journal.pcbi.1003542.

Jones, C., C. Blanchette, M. Brooke, J. Harris, M. Jones, and M. Schildhauer. 2007. “A Metadata-Driven Framework for Generating Field Data Entry Interfaces in Ecology.” Ecological Informatics 2 (3): 270–78. https://doi.org/10.1016/j.ecoinf.2007.06.005.

Lowndes, J. S. S., B. D. Best, C. Scarborough, J. C. Afflerbach, M. R. Frazier, C. C. O’Hara, N. Jiang, and B. S. Halpern. 2017. “Our Path to Better Science in Less Time Using Open Data Science Tools.” Nature Ecology & Evolution 1 (0160): 1–7. https://doi.org/10.1038/s41559-017-0160.

Michener, W. K. 2015. “Ten Simple Rules for Creating a Good Data Management Plan.” PLoS Computational Biology 11 (10): e1004525. https://doi.org/10.1371/journal.pcbi.1004525.

Sherwood, E. T., G. Raulerson, M. W. Beck, and M. Burke. 2020. “Tampa Bay Estuary Program: Quality Management Plan.” 16-20. St. Petersburg, Florida: Tampa Bay Estuary Program. https://drive.google.com/file/d/1DyA0PNHV8rEXGMwGiyS7sXY1ECLYpJJO/view?usp=sharing.

Wickham, H., and J. Bryan. 2015. R Packages. Sebastopol, California: O’Reilly.

Wickham, H., and J. Hesselberth. 2020. Pkgdown: Make Static Html Documentation for a Package. https://CRAN.R-project.org/package=pkgdown.


text:
## Electrical theory

Calculate the rms value of the half‐wave rectified sinusoidal wave shown in the Figure below.

Hint: For the integration use the trigonometric identity cos2x=1/2 (1+cos2x).

format: plain


text:
Dedekind Cuts in Rudin' analysis - Step 4

This question refers to the construction of $\mathbb{R}$ from $\mathbb{Q}$ using Dedekind cuts, as presented in Rudin's "Principles of Mathematical Analysis" pp. 17-21.

More specifically, in the last paragraph of step 4, Rudin says that for $\alpha$ a fixed cut, and given $v \in 0^*$, setting $w=- v / 2$, there exists an integer $n$ such that $nw \in \alpha$ but $(n+1)w$ is not inside $\alpha$. Rudin says that this depends on the Archimedean property of the rationals, however he has not proved it.

Could somebody prove the existence of the integer $n$?

By definition there is some rational $q$ that is not in $\alpha$. Let $p=|q|+1$; then $p>q$, so $p\notin\alpha$, and $p>0$. The rational $w$ is also positive, so by the Archimedean property of the rationals there is a positive integer $m$ such that $mw>p$, and therefore $mw\notin\alpha$.

We also know that there is some rational $r\in\alpha$. Let $s=-|r|-1$; then $s<r$, so $s\in\alpha$, and $-s>0$, so again by the Archimedean property there is a positive integer $k$ such that $kw>-s$. But then $(-k)w<s$, so $(-k)w\in\alpha$. We now have positive integers $k$ and $m$ such that $-kw\in\alpha$ and $mw\notin\alpha$.

Let $S=\{i\in\Bbb N:(i-k)w\notin\alpha\}$; $m+k\in S$, so $S\ne\varnothing$. Since $\Bbb N$ is well-ordered, $S$ has a smallest element, $j$. Note that $0\notin S$, since $-kw\in\alpha$, so $j>0$. Let $n=(j-1)-k$. Then $nw\in\alpha$, and $(n+1)w\notin\alpha$.

• Concerning the last part... I've checked, and think Rudin does not mention that $\mathbb N$ is well-ordered in chapter 1 or anywhere else in the book. Might this have been his omission, or is the well-ordering of $\mathbb N$ a fact the reader is supposed to know before reading Rudin? – Anthony Pulido Dec 1 '14 at 20:23
• @Anthony: Probably the latter: Walter was extremely careful about such things. Still, I’d not bet a large amount either way. – Brian M. Scott Dec 1 '14 at 20:25
• @Brian, I have seen a few proofs on Math StackExchange for proving that the rationals have the Archimedean property, the simplest being basically this: Let $m/n \in Q$. Then $n(m/n) + 1 \gt m/n$ (math.stackexchange.com/q/61632). I'm wondering if this is really rigorous, and if there are any underlying assumptions that I am missing. I am working through Rudin's construction of $R$ using cuts right now as well. – addisongpi Oct 2 '16 at 16:27
• @Addison: It’s rigorous as the term is normally used. To make it any more rigorous, one would have to go back to the construction of the rationals and the verification that they are an ordered field and recast the argument in that context, verifying that it still works. I’d say that you’re probably not missing anything. – Brian M. Scott Oct 2 '16 at 19:01

If $v\in 0^*$, then $v<0$, so $w>0$. Let $\gamma=\sup \alpha$. The Archimedean property says that for any $\gamma$ there is some integer $m$ such that $mw\geq\gamma$. Since $\mathbb{N}$ is well-ordered, there is a smallest $m$ such that $mw\geq\gamma$, call this $n+1$. This means that $nw<\gamma$, and hence $nw\in \alpha$. But also $(n+1)w\notin \alpha$ since it is larger than $\gamma$ (OK, so there might be some subtleties with endpoints you should think about, but essentially this is the idea).

• How do you know that $\gamma=\sup \alpha$ exists? – Manos Aug 26 '12 at 17:10
• Sorry. I don't have the book in front of me, but isn't this step 3? – Matt Aug 26 '12 at 18:26
• As i understand $\alpha$ is a cut, and as such it has no upper bound. – Manos Aug 26 '12 at 20:25
• @ Matt: I think step 3 proves that $R$, the set of cuts, has the least upper bound property, not that a cut has a least upper bound, which it does not have to have since it is a subset of the rationals. Supposing you were right about that, I'm also not sure of what follows in your argument. Suppose the cut $\alpha$ represents $-20$, so that $\gamma = -20$, and say, $v = -2 \in 0^*$, so that $w = -v/2 = 1$. (continued.) – Anthony Pulido Dec 1 '14 at 23:24
• The archimedean property says that there is a positive integer $m$ such that $mw > \gamma$, or $m > -20$, as you said. Just to be sure, by the second fact to "be used freely" (written just before step 2): if $r \notin \alpha$ and $r < s$ then $s \notin \alpha$, it follows that $m \notin \alpha$, although we already knew this since $m$ is bigger than the supremum. Since $m \in \mathbb N$ and you are using its well-ordering, the smallest possible value for $m$ is $1$. If $m = n + 1$, then $n = 0$. It follows that $nw = 0 \notin \alpha$ and $(n+1)w \notin \alpha$. – Anthony Pulido Dec 1 '14 at 23:25


text:
1.9k views

The following function computes the value of  $\binom{m}{n}$ correctly for all legal values $m$ and $n$ ($m ≥1, n ≥ 0$ and $m > n$)

int func(int m, int n)
{
if (E) return 1;
else return(func(m -1, n) + func(m - 1, n - 1));
}

In the above function, which of the following is the correct expression for E?

1. $(n = = 0) || (m = = 1)$
2. $(n = = 0)$ && $(m = = 1)$
3. $(n = = 0) || (m = = n)$
4. $(n = = 0)$ && $(m = = n)$

edited | 1.9k views
0

Computational complexity is O($2^{m}$).

Notice answer( expression E) is dependent on conditions (m≥1,n≥0 and m>n). If these conditions are changes then answer(expression E) may change.
+4

mC0 = 1
mC= 1

Hence, (C) is the correct option!

0
What will be the recurrence relation for this? Anyone please help.
+1
@Shubhgupta, Recurrence Relation  is simply $T(m,n) = T(m-1,n)+T(m-1,n-1)$ with base conditions $T(n,n)=1$ , $T(m,0)=1$. It is a Recurrence Relation with $2$ variables.
+1

Thanks @ankitgupta.1729 !

Because $\binom{m}{0}$ = $1$ and $\binom{n}{n}$ = $1$.

by Boss (33.8k points)
edited
+2
I got its ans by substituting some values, but not able to understand how this recursion is giving C(m,n).
@Arjun sir , please spread some light on it.
+4

@vijaycs

We know by drawing pascals triangle we can find C(m,n).

And given recursion leads to pascal triangle.

I think following link may address ur concern.

https://en.wikipedia.org/wiki/Binomial_coefficient#Recursive_formula

https://en.wikipedia.org/wiki/Pascal%27s_triangle

+4
@vijay where is problem

Say

we are finding value $_{2}^{4}\textrm{C}$=$_{2}^{3}\textrm{C}$+$_{1}^{3}\textrm{C}$

@Rajesh it is just finding combination
+4
@Srestha, I did not get you . . .

$nC_{r} = (n-1)C_{r} + (n-1)C_{r-1}\\ = \frac{(n-1)!}{r!(n-1-r)!} + \frac{(n-1)!}{(r-1)!(n-r)!}\\ = \frac{(n-1)!}{(r-1)!(n-1-r)!}.[\frac{1}{r} - \frac{1}{(n-r)}]\\ = \frac{(n-1)!}{(r-1)!(n-1-r)!}.\frac{n}{r(n-r)}\\ =\frac{(n)!}{(r)!(n-r)!}$
0
and @vijay  i did not get u why u r showing that one
+3
It is the proof of the question.
0
why option A is incorrect.

let func(2,1)   // m=2, n=1

func(2,1) = func(1,1)  + func(1,0)

in this case option A & option C both gives correct ans.

Can anyone give a counter example for option A??
+3
Option (a) is incorrect.

According to option (a) :: func(m,n)=1 if n==0 or m==1

Now,

f(3,3)=f(2,3) + f(2,2)

f(2,3)= f(1,3) + f(1,2) = 1+1=2 //here m=1

f(2,2)=f(1,2)+f(1,1)= 1+1 = 2

So, f(3,3)=4 ...This is wrong

as f(3,3)=1 and not 4.Hence option (c) correct
0
Can anyone confirm time complexity ?

O(2^m) ?
+2
@VS
From m & n, m is a variable & n should be a constant.
I guess T(m) =2T(m-1)+c
so seems as O(2^m)
+1
@Ahwan ur recurrence relation is fine but you cannot say that n is a constant
0
@VS I guess you cant find the recurrence relation otherwise..  if n is not a constant.
+1
@Ahwan ,but how can you say that n is a constant ? We are computing mCn.

And regarding the recurrence relation.

See if we try to make a Recurrence tree for it. We will surely m-levels but it will not be a complete BT because of some gaps caused by n.

But if we try to appoximate ..Recurrence relation would be that only and TC=2^m
+1
@VS  yes if  n is a variable then the recurrence relation I have given is not correct.
By the way time complexity with multiple variables is complex.

0

It's base case need to be taken care.

Return 1 when you reach base case

which is

mCm or mC0

0
why not D? && means we have to check for both conditions to be true isnt?
0

@Pranav Because if(E) means if(E!=0); so if any one of them is FALSE then we come out of the loop.

ans..(C).
by Active (5k points)

1
2


text:
Home / Arithmetic Aptitude / Pipes and Cistern :: General Questions

### Arithmetic Aptitude :: Pipes and Cistern

1. Three pipes A, B and C can fill a tank from empty to full in 30 minutes, 20 minutes, and 10 minutes respectively. When the tank is empty, all the three pipes are opened. A, B and C discharge chemical solutions P,Q and R respectively. What is the proportion of the solution R in the liquid in the tank after 3 minutes?

2.  A. $$\frac { 5 } { 11 }$$ B. $$\frac {6 } { 11 }$$ C. $$\frac {7 } { 11 }$$ D. $$\frac {8 } { 11 }$$

3. Pipes A and B can fill a tank in 5 and 6 hours respectively. Pipe C can empty it in 12 hours. If all the three pipes are opened together, then the tank will be filled in:

4.  A. 1 $$\frac { 13 } { 17 }$$hours B. 2 $$\frac { 8 } { 11 }$$hours C. 3 $$\frac { 9 } { 17 }$$hours D. 4$$\frac { 1} { 2 }$$ hours

5.  A pump can fill a tank with water in 2 hours. Because of a leak, it took 2 $$\frac { 1 } { 3}$$ hours to fill the tank. The leak can drain all the water of the tank in:
6.  A. 4$$\frac { 1 } { 3}$$hours B. 7 hours C. 8 hours D. 14 hours

7. Two pipes A and B can fill a cistern in 3 $$\frac { 1 } { 2 }$$ minutes and 45 minutes respectively. Both pipes are opened. The cistern will be filled in just half an hour, if the B is turned off after:

8.  A. 5 min. B. 9 min. C. 10 min. D. 15 min.

9.  A tank is filled by three pipes with uniform flow. The first two pipes operating simultaneously fill the tank in the same time during which the tank is filled by the third pipe alone. The second pipe fills the tank 5 hours faster than the first pipe and 4 hours slower than the third pipe. The time required by the first pipe is:
10.  A. 6 hours B. 10 hours C. 15 hours D. 30 hours

11. Two pipes can fill a tank in 20 and 24 minutes respectively and a waste pipe can empty 3 gallons per minute. All the three pipes working together can fill the tank in 15 minutes. The capacity of the tank is:

12.  A. 60 gallons B. 100 gallons C. 120 gallons D. 180 gallons

13. A tank is filled in 5 hours by three pipes A, B and C. The pipe C is twice as fast as B and B is twice as fast as A. How much time will pipe A alone take to fill the tank?

14.  A. 20 hours B. 25 hours C. 35 hours D. Cannot be determined E. None of these

15.  Two pipes A and B together can fill a cistern in 4 hours. Had they been opened separately, then B would have taken 6 hours more than A to fill the cistern. How much time will be taken by A to fill the cistern separately?
16.  A. 1 hour B. 2 hours C. 6 hours D. 8 hours

17. Two pipes A and B can fill a tank in 20 and 30 minutes respectively. If both the pipes are used together, then how long will it take to fill the tank?

18.  A. 12 min B. 15 min C. 25 min D. 50 min

19. Two pipes A and B can fill a tank in 15 minutes and 20 minutes respectively. Both the pipes are opened together but after 4 minutes, pipe A is turned off. What is the total time required to fill the tank?

20.  A. 10 min. 20 sec. B. 11 min. 45 sec. C. 12 min. 30 sec. D. 14 min. 40 sec.


text:
If the Grothendieck ring of a semiring on a free commutative monoid is unital, is the original semiring unital?

Suppose $$S$$ is an associative semiring whose underling commutative monoid is free (in particular, cancellative) and that its Grothendieck ring $$G(S)$$ is a unital ring. Can we conclude that $$S$$ must be unital, and if not, is there a nice counter-example?

Alternatively, are there additional suppositions we can put on $$S$$ which would allow us to conclude the unitality of $$S$$ from that of $$G(S)$$?

The context is that I am struggling to disprove that a particular ring I have constructed is unital, and I have proven it is of the form $$G(S)$$ for $$S$$ which is not unital. I am hoping this helps me somehow.

Cross-posted on math.stackexchange.

The answer is no. Let $$S$$ be a finite meet semilattice without maximum. For concreteness, take $$S$$ to be the proper subsets of $$\{1,2\}$$ under intersection. Let $$\mathbb NS$$ be the semigroup semiring of $$S$$. Then the underlying additive monoid is free on $$S$$. The Grothendieck ring is the semigroup ring $$\mathbb ZS$$ and this ring has an identity, but the identity cannot be expressed as a non-negative linear combination of elements of $$S$$. One writes it using inclusion-exclusion (or Mobius inversion).

For my particular $$S$$, the proper subsets of $$\{1,2\}$$, the identity element of $$\mathbb ZS$$ is $$e=\{1\}+\{2\}-\emptyset$$, which does not belong to $$\mathbb NS$$. Recalling that my semigroup multiplication is intersection, you can just check that $$e$$ multiplied by an element of $$S$$ returns that element.

For a more general meet semilattice $$S$$, the identity is $$\sum_{s\in S}\sum_{t\leq s}t\mu(t,s)$$ where $$\mu$$ is the Mobius function of $$S$$.

I think for general semirings there is no hope of a reasonable condition. For example if $$S$$ is a finite left regular band (that is a semigroup satisfying the identities $$xyx=xy$$ and $$x^2=x$$), I showed with Margolis and Saliola that $$\mathbb ZS$$ has an identity if and only if certain posets associated to $$S$$ have connected Hasse diagrams. But if $$S$$ is not a monoid, the identity will not belong to $$\mathbb NS$$. This is already a hard result. The meet semilattice case is just a commutative left regular band.

I also showed that for any finite inverse semigroup $$S$$, the semigroup ring $$\mathbb ZS$$ has an identity, but $$\mathbb NS$$ will not have an identity unless $$S$$ has an identity. This also generalizes the meet semilattice case.

Update: Here are some more observations. First, if $$S$$ is a semigroup, then the semigroup ring $$\mathbb NS$$ is unital if and only if $$S$$ is a monoid. That is because you have the surjective augmentation homomorphism $$\mathbb NS\to\mathbb N$$ summing up the coefficients and since the only nonzero idempotent of $$\mathbb N$$ is $$1$$, we must have that the support of an identity element has one element of $$S$$ with coefficient $$1$$ and so $$S$$ is a monoid.

Second, let $$S$$ be a meet semilattice (possibly infinite) viewed as a semigroup under meet. Then $$\mathbb NS$$ has an identity iff $$S$$ has a maximum. But $$\mathbb ZS$$ is isomorphic to the locally constant functions with compact support from $$\widehat{S}\to \mathbb Z$$ under pointwise operations. Here $$\widehat{S}$$ is the space of all non-zero semigroup homomorphisms $$\phi\colon S\to \{0,1\}$$ with the topology of pointwise convergence, where $$\{0,1\}$$ is a semigroup under usual multiplication. Thus $$\mathbb ZS$$ has an identity if and only if $$\widehat{S}$$ is compact. This is obviously a topological issue.

• This is incredibly enlightening, thank you! Do you have a reference to your result about when $\mathbb{Z}S$ has an identity? I don’t quite have a semigroup semiring, but this sounds very interesting to me. I quickly looked but didn’t find it. May 19 at 17:49
• Which result did you want a reference to? I mentioned several. May 19 at 18:21
• The case of a finite meet semilattice can be extracted from Theorem 1 of Solomon's paper sciencedirect.com/science/article/pii/S0021980067800644 and its proof May 19 at 18:28
• Also you might look at 4.10 and 4.15 of arxiv.org/pdf/1508.05446.pdf#page72 May 19 at 19:03


text:
• We present ultraviolet (UV) spectroscopy and photometry of four Type Ia supernovae (SNe 2004dt, 2004ef, 2005M, and 2005cf) obtained with the UV prism of the Advanced Camera for Surveys on the Hubble Space Telescope. This dataset provides unique spectral time series down to 2000 Angstrom. Significant diversity is seen in the near maximum-light spectra (~ 2000--3500 Angstrom) for this small sample. The corresponding photometric data, together with archival data from Swift Ultraviolet/Optical Telescope observations, provide further evidence of increased dispersion in the UV emission with respect to the optical. The peak luminosities measured in uvw1/F250W are found to correlate with the B-band light-curve shape parameter dm15(B), but with much larger scatter relative to the correlation in the broad-band B band (e.g., ~0.4 mag versus ~0.2 mag for those with 0.8 < dm15 < 1.7 mag). SN 2004dt is found as an outlier of this correlation (at > 3 sigma), being brighter than normal SNe Ia such as SN 2005cf by ~0.9 mag and ~2.0 mag in the uvw1/F250W and uvm2/F220W filters, respectively. We show that different progenitor metallicity or line-expansion velocities alone cannot explain such a large discrepancy. Viewing-angle effects, such as due to an asymmetric explosion, may have a significant influence on the flux emitted in the UV region. Detailed modeling is needed to disentangle and quantify the above effects.
• ### An Unbiased Method of Modeling the Local Peculiar Velocity Field with Type Ia Supernovae(1103.1603)

March 8, 2011 astro-ph.CO
We apply statistically rigorous methods of nonparametric risk estimation to the problem of inferring the local peculiar velocity field from nearby supernovae (SNIa). We use two nonparametric methods - Weighted Least Squares (WLS) and Coefficient Unbiased (CU) - both of which employ spherical harmonics to model the field and use the estimated risk to determine at which multipole to truncate the series. We show that if the data are not drawn from a uniform distribution or if there is power beyond the maximum multipole in the regression, a bias is introduced on the coefficients using WLS. CU estimates the coefficients without this bias by including the sampling density making the coefficients more accurate but not necessarily modeling the velocity field more accurately. After applying nonparametric risk estimation to SNIa data, we find that there are not enough data at this time to measure power beyond the dipole. The WLS Local Group bulk flow is moving at 538 +- 86 km/s towards (l,b) = (258 +- 10 deg, 36 +- 11 deg) and the CU bulk flow is moving at 446 +- 101 km/s towards (l,b) = (273 +- 11 deg, 46 +- 8 deg). We find that the magnitude and direction of these measurements are in agreement with each other and previous results in the literature.
• ### The Carnegie Supernova Project: First Near-Infrared Hubble Diagram to z~0.7(0907.4524)

July 26, 2009 astro-ph.CO
The Carnegie Supernova Project (CSP) is designed to measure the luminosity distance for Type Ia supernovae (SNe Ia) as a function of redshift, and to set observational constraints on the dark energy contribution to the total energy content of the Universe. The CSP differs from other projects to date in its goal of providing an I-band {rest-frame} Hubble diagram. Here we present the first results from near-infrared (NIR) observations obtained using the Magellan Baade telescope for SNe Ia with 0.1 < z < 0.7. We combine these results with those from the low-redshift CSP at z <0.1 (Folatelli et al. 2009). We present light curves and an I-band Hubble diagram for this first sample of 35 SNe Ia and we compare these data to 21 new SNe Ia at low redshift. These data support the conclusion that the expansion of the Universe is accelerating. When combined with independent results from baryon acoustic oscillations (Eisenstein et al. 2005), these data yield Omega_m = 0.27 +/- 0.0 (statistical), and Omega_DE = 0.76 +/- 0.13 (statistical) +/- 0.09 (systematic), for the matter and dark energy densities, respectively. If we parameterize the data in terms of an equation of state, w, assume a flat geometry, and combine with baryon acoustic oscillations, we find that w = -1.05 +/- 0.13 (statistical) +/- 0.09 (systematic). The largest source of systematic uncertainty on w arises from uncertainties in the photometric calibration, signaling the importance of securing more accurate photometric calibrations for future supernova cosmology programs. Finally, we conclude that either the dust affecting the luminosities of SNe Ia has a different extinction law (R_V = 1.8) than that in the Milky Way (where R_V = 3.1), or that there is an additional intrinsic color term with luminosity for SNe Ia independent of the decline rate.
• ### Educating the Next Generation of Leading Scientists: Turning Ideas into Action(0903.3327)

March 19, 2009 astro-ph.IM
The core of scientific research is turning new ideas into reality. From the school science fair to the search for the secrets of dark energy, high-quality research consists of scientific investigation constrained within the scope of a well-defined project. Large or small, generously funded or just scraping by,scientific projects use time, money, and information to turn ideas into plans, plans into action, and action into results. While we, as a community, do much to educate students in the techniques of research, we do not systematically train students in the nature and organization of scientific projects or in the techniques of project management. We propose a two-pronged attack to address this issue in the next decade. First, to generate a broad base of future scientists who have a basic familiarity with the ideas of projects, we propose that the community develop standards for the content of a project design and management course in astronomy and astrophysics. Second, to train future scientists to assume leadership roles in new investigations in astronomy and astrophysics, we propose that the community develop standards for graduate programs in the area of research project leadership.
• ### Exploring Dark Energy with Next-Generation Photometric Redshift Surveys(0902.2599)

Feb. 16, 2009 astro-ph.CO
The coming decade will be an exciting period for dark energy research, during which astronomers will address the question of what drives the accelerated cosmic expansion as first revealed by type Ia supernova (SN) distances, and confirmed by later observations. The mystery of dark energy poses a challenge of such magnitude that, as stated by the Dark Energy Task Force (DETF), "nothing short of a revolution in our understanding of fundamental physics will be required to achieve a full understanding of the cosmic acceleration." The lack of multiple complementary precision observations is a major obstacle in developing lines of attack for dark energy theory. This lack is precisely what next-generation surveys will address via the powerful techniques of weak lensing (WL) and baryon acoustic oscillations (BAO) -- galaxy correlations more generally -- in addition to SNe, cluster counts, and other probes of geometry and growth of structure. Because of their unprecedented statistical power, these surveys demand an accurate understanding of the observables and tight control of systematics. This white paper highlights the opportunities, approaches, prospects, and challenges relevant to dark energy studies with wide-deep multiwavelength photometric redshift surveys. Quantitative predictions are presented for a 20000 sq. deg. ground-based 6-band (ugrizy) survey with 5-sigma depth of r~27.5, i.e., a Stage 4 survey as defined by the DETF.
• ### Survey Requirements for Accurate and Precise Photometric Redshifts for Type Ia Supernovae(0708.0033)

Aug. 21, 2007 astro-ph
In this paper we advance the simple analytic photometric redshift estimator for Type Ia supernovae (SNe Ia) proposed by Wang (2007), and use it to study simulated SN Ia data. We find that better than 0.5% accuracy in z_phot (with RMS[(z_phot-z_spec)/(1+z_spec)]<0.005) is possible for SNe Ia with well sampled lightcurves in three observed passbands (riz) with a signal-to-noise ratio of 25 at peak brightness, if the extinction by dust is negligible. The corresponding bias in z_phot (the mean of (z_phot-z_spec)) is 5.4\times 10^{-4}. If dust extinction is taken into consideration in the riz observer-frame lightcurves, the accuracy in z_phot deteriorates to 4.4%, with a bias in z_phot of 8.0\times 10^{-3}. Adding the g band lightcurve improves the accuracy in z_phot to 2.5%, and reduces the bias in z_phot to -1.5\times 10^{-3}. Our results have significant implications for the design of future photometric surveys of SNe Ia from both ground and space telescopes. Accurate and precise photometric redshifts boost the cosmological utility of such surveys.
• ### Multicolor Infrared Observations of SN 2006aj, the Supernova Associated with XRF 060218 - Paper I(astro-ph/0612621)

March 13, 2007 astro-ph
We report simultaneous multicolor near-infrared (NIR) observations of the supernova associated with x-ray Flash 060218 during the first 16 days after the high energy event. We find that the light curve rises and peaks relatively fast compared to other SN Ic, with the characteristic broad NIR peak seen in all three bands. We find that the rise profile before the peak is largely independent of NIR wavelength, each band appearing to transition into a plateau phase around day 10--13. Since the light curve is in the plateau phase when our observations end at day 16, we can only place limits on the peak absolute magnitudes, but we estimate that SN 2006aj is one of the lowest NIR luminosity XRF/GRB associated SNe observed to date. The broad peaks observed in the {\em JHK$_s$} bands point to a large increase in the NIR contribution of the total flux output from days 10--16. This evolution can be seen in the broad color and SED diagrams constructed using {\em UBVRIJHK$_s$} monochromatic flux measurements for the first 16 days of the event. Ultimately, a 10-day rise time would make SN 2006aj an extremely fast rise SN Ic event, faster than SN 1998bw and SN 2003dh, which combined with its underluminous nature, indicates a lower amount of $^{56}$Ni ejected by the progenitor compared to other XRF/GRB-SNe. Furthermore, the lack of significant color change during the rise portion of the burst points to little or no spectral evolution over the first 10 days of activity in the NIR.
• ### Supernova Cosmology and the ESSENCE project(astro-ph/0510026)

Oct. 2, 2005 astro-ph
The proper usage of Type Ia supernovae (SNe Ia) as distance indicators has revolutionized cosmology, and added a new dominant component to the energy density of the Universe, dark energy. Following the discovery and confirmation era, the currently ongoing SNe Ia surveys aim to determine the properties of the dark energy. ESSENCE is a five year ground-based supernova survey aimed at finding and characterizing 200 SNe Ia in the redshift domain z=[0.2-0.8]. The goal of the project is to put constraints on the equation of state parameter, w, of the dark energy with an accuracy of <10%. This paper presents these ongoing efforts in the context of the current developments in observational cosmology.