text:
• Galaxy and Mass Assembly (GAMA): Variation in Galaxy Structure Across the Green Valley(1804.04470)

April 12, 2018 astro-ph.GA
Using a sample of 472 local Universe (z<0.06) galaxies in the stellar mass range 10.25 < log M*/M_sun < 10.75, we explore the variation in galaxy structure as a function of morphology and galaxy colour. Our sample of galaxies is sub-divided into red, green and blue colour groups and into elliptical and non-elliptical (disk-type) morphologies. Using KiDS and VIKING derived postage stamp images, a group of eight volunteers visually classified bars, rings, morphological lenses, tidal streams, shells and signs of merger activity for all systems. We find a significant surplus of rings ($2.3\sigma$) and lenses ($2.9\sigma$) in disk-type galaxies as they transition across the green valley. Combined, this implies a joint ring/lens green valley surplus significance of $3.3\sigma$ relative to equivalent disk-types within either the blue cloud or the red sequence. We recover a bar fraction of ~44% which remains flat with colour, however, we find that the presence of a bar acts to modulate the incidence of rings and (to a lesser extent) lenses, with rings in barred disk-type galaxies more common by ~20-30 percentage points relative to their unbarred counterparts, regardless of colour. Additionally, green valley disk-type galaxies with a bar exhibit a significant $3.0\sigma$ surplus of lenses relative to their blue/red analogues. The existence of such structures rules out violent transformative events as the primary end-of-life evolutionary mechanism, with a more passive scenario the favoured candidate for the majority of galaxies rapidly transitioning across the green valley.
• The Taipan Galaxy Survey: Scientific Goals and Observing Strategy(1706.01246)

March 19, 2018 astro-ph.CO, astro-ph.GA
Taipan is a multi-object spectroscopic galaxy survey starting in 2017 that will cover 2pi steradians over the southern sky, and obtain optical spectra for about two million galaxies out to z<0.4. Taipan will use the newly-refurbished 1.2m UK Schmidt Telescope at Siding Spring Observatory with the new TAIPAN instrument, which includes an innovative 'Starbugs' positioning system capable of rapidly and simultaneously deploying up to 150 spectroscopic fibres (and up to 300 with a proposed upgrade) over the 6-deg diameter focal plane, and a purpose-built spectrograph operating from 370 to 870nm with resolving power R>2000. The main scientific goals of Taipan are: (i) to measure the distance scale of the Universe (primarily governed by the local expansion rate, H_0) to 1% precision, and the structure growth rate of structure to 5%; (ii) to make the most extensive map yet constructed of the mass distribution and motions in the local Universe, using peculiar velocities based on improved Fundamental Plane distances, which will enable sensitive tests of gravitational physics; and (iii) to deliver a legacy sample of low-redshift galaxies as a unique laboratory for studying galaxy evolution as a function of mass and environment. The final survey, which will be completed within 5 years, will consist of a complete magnitude-limited sample (i<17) of about 1.2x10^6 galaxies, supplemented by an extension to higher redshifts and fainter magnitudes (i<18.1) of a luminous red galaxy sample of about 0.8x10^6 galaxies. Observations and data processing will be carried out remotely and in a fully-automated way, using a purpose-built automated 'virtual observer' software and an automated data reduction pipeline. The Taipan survey is deliberately designed to maximise its legacy value, by complementing and enhancing current and planned surveys of the southern sky at wavelengths from the optical to the radio.
• Jeans that fit: weighing the mass of the Milky Way analogues in the $\Lambda{\rm CDM}$ universe(1801.03950)

Jan. 15, 2018 astro-ph.CO, astro-ph.GA
The spherical Jeans equation is a widely used tool for dynamical study of gravitating systems in astronomy. Here we test its efficacy in robustly weighing the mass of Milky Way analogues, given they need not be in equilibrium or even spherical. Utilizing Milky Way stellar halos simulated in accordance with $\Lambda{\rm CDM}$ cosmology by Bullock and Johnston (2005) and analysing them under the Jeans formalism, we recover the underlying mass distribution of the parent galaxy, within distance $r/{\rm kpc}\in[10,100]$, with a bias of $\sim12\%$ and a dispersion of $\sim14\%$. Additionally, the mass profiles of triaxial dark matter halos taken from the SURFS simulation, within scaled radius $0.2<r/r_{\rm max}<3$, are measured with a bias of $\sim-2.4\%$ and a dispersion of $\sim10\%$. The obtained dispersion is not because of Poisson noise due to small particle numbers as it is twice the later. We interpret the dispersion to be due to the inherent nature of the $\Lambda{\rm CDM}$ halos, for example being aspherical and out-of-equilibrium. Hence the dispersion obtained for stellar halos sets a limit of about $12\%$ (after adjusting for random uncertainty) on the accuracy with which the mass profiles of the Milky Way-like galaxies can be reconstructed using the spherical Jeans equation. This limit is independent of the quantity and quality of the observational data. The reason for a non zero bias is not clear, hence its interpretation is not obvious at this stage.
• The SAMI Galaxy Survey: Spatially Resolving the Main Sequence of Star Formation(1801.04283)

Jan. 12, 2018 astro-ph.GA
We present the ~800 star formation rate maps for the SAMI Galaxy Survey based on H{\alpha} emission maps, corrected for dust attenuation via the Balmer decrement, that are included in the SAMI Public Data Release 1. We mask out spaxels contaminated by non-stellar emission using the [O III]/H{\beta}, [N II]/H{\alpha}, [S II]/H{\alpha}, and [O I]/H{\alpha} line ratios. Using these maps, we examine the global and resolved star-forming main sequences of SAMI galaxies as a function of morphology, environmental density, and stellar mass. Galaxies further below the star-forming main sequence are more likely to have flatter star formation profiles. Early-type galaxies split into two populations with similar stellar masses and central stellar mass surface densities. The main sequence population has centrally-concentrated star formation similar to late-type galaxies, while galaxies >3{\sigma} below the main sequence show significantly reduced star formation most strikingly in the nuclear regions. The split populations support a two-step quenching mechanism, wherein halo mass first cuts off the gas supply and remaining gas continues to form stars until the local stellar mass surface density can stabilize the reduced remaining fuel against further star formation. Across all morphologies, galaxies in denser environments show a decreased specific star formation rate from the outside in, supporting an environmental cause for quenching, such as ram-pressure stripping or galaxy interactions.
• The Need for Speed: Escape velocity and dynamical mass measurements of the Andromeda galaxy(1801.03949)

Jan. 11, 2018 astro-ph.CO, astro-ph.GA
Our nearest large cosmological neighbour, the Andromeda galaxy (M31), is a dynamical system, and an accurate measurement of its total mass is central to our understanding of its assembly history, the life-cycles of its satellite galaxies, and its role in shaping the Local Group environment. Here, we apply a novel approach to determine the dynamical mass of M31 using high velocity Planetary Nebulae (PNe), establishing a hierarchical Bayesian model united with a scheme to capture potential outliers and marginalize over tracers unknown distances. With this, we derive the escape velocity run of M31 as a function of galacto-centric distance, with both parametric and non-parametric approaches. We determine the escape velocity of M31 to be $470\pm{40}$ km s$^{-1}$ at a galacto-centric distance of 15 kpc, and also, derive the total potential of M31, estimating the virial mass and radius of the galaxy to be $0.8\pm{0.1}\times10^{12}\,M_\odot$ and $240\pm{10}$ kpc, respectively. Our M31 mass is on the low-side of the measured range, this supports the lower expected mass of the M31-Milky Way system from the timing and momentum arguments, satisfying the HI constraint on circular velocity between $10\lesssim R/\textrm{kpc}<35$, and agreeing with the stellar mass Tully-Fisher relation. To place these results in a broader context, we compare them to the key predictions of the $\Lambda{\rm CDM}$ cosmological paradigm, including the stellar-mass-halo-mass and the dark matter halo concentration-virial mass correlation, and finding it to be an outlier to this relation.
• On the observability of individual Population III stars and their stellar-mass black hole accretion disks through cluster caustic transits(1801.03584)

Jan. 10, 2018 astro-ph.CO, astro-ph.GA
We summarize panchromatic Extragalactic Background Light data to place upper limits on the integrated near-infrared surface brightness (SB) that may come from Population III stars and possible accretion disks around their stellar-mass black holes (BHs) in the epoch of First Light, broadly taken from z$\simeq$7-17. Theoretical predictions and recent near-infrared power-spectra provide tighter constraints on their sky-signal. We outline the physical properties of zero metallicity Population III stars from MESA stellar evolution models through helium-depletion and of BH accretion disks at z$\gtrsim$7. We assume that second-generation non-zero metallicity stars can form at higher multiplicity, so that BH accretion disks may be fed by Roche-lobe overflow from lower-mass companions. We use these near-infrared SB constraints to calculate the number of caustic transits behind lensing clusters that the James Webb Space Telescope and the next generation ground-based telescopes may observe for both Population III stars and their BH accretion disks. Typical caustic magnifications can be $\mu$$\simeq10^4-10^5, with rise times of hours and decline times of \lesssim1 year for cluster transverse velocities of v_{T}$$\lesssim$1000 km s$^{-1}$. Microlensing by intracluster medium objects can modify transit magnifications, but lengthen visibility times. Depending on BH masses, accretion-disk radii and feeding efficiencies, stellar-mass BH accretion-disk caustic transits could outnumber those from Population III stars. To observe Population III caustic transits directly may require to monitor 3-30 lensing clusters to AB$\lesssim$29 mag over a decade.
• GAMA/G10-COSMOS/3D-HST: The 0<z<5 cosmic star-formation history, stellar- and dust-mass densities(1710.06628)

Oct. 19, 2017 astro-ph.CO, astro-ph.GA
We use the energy-balance code MAGPHYS to determine stellar and dust masses, and dust corrected star-formation rates for over 200,000 GAMA galaxies, 170,000 G10-COSMOS galaxies and 200,000 3D-HST galaxies. Our values agree well with previously reported measurements and constitute a representative and homogeneous dataset spanning a broad range in stellar mass (10^8---10^12 Msol), dust mass (10^6---10^9 Msol), and star-formation rates (0.01---100 Msol per yr), and over a broad redshift range (0.0 < z < 5.0). We combine these data to measure the cosmic star-formation history (CSFH), the stellar-mass density (SMD), and the dust-mass density (DMD) over a 12 Gyr timeline. The data mostly agree with previous estimates, where they exist, and provide a quasi-homogeneous dataset using consistent mass and star-formation estimators with consistent underlying assumptions over the full time range. As a consequence our formal errors are significantly reduced when compared to the historic literature. Integrating our cosmic star-formation history we precisely reproduce the stellar-mass density with an ISM replenishment factor of 0.50 +/- 0.07, consistent with our choice of Chabrier IMF plus some modest amount of stripped stellar mass. Exploring the cosmic dust density evolution, we find a gradual increase in dust density with lookback time. We build a simple phenomenological model from the CSFH to account for the dust mass evolution, and infer two key conclusions: (1) For every unit of stellar mass which is formed 0.0065---0.004 units of dust mass is also formed; (2) Over the history of the Universe approximately 90 to 95 per cent of all dust formed has been destroyed and/or ejected.
• The abundance of ultra-diffuse galaxies from groups to clusters: UDGs are relatively more common in more massive haloes(1706.02704)

Aug. 11, 2017 astro-ph.GA
In recent years, multiple studies have reported substantial populations of large, low-surface-brightness galaxies in local galaxy clusters. Various theories that aim to explain the presence of such ultra-diffuse galaxies (UDGs) have since been proposed. A key question that will help to differentiate between models is whether UDGs have counterparts in lower-mass host haloes, and what their abundance as a function of halo mass is. In this study we extend our previous study of UDGs in galaxy clusters to galaxy groups. We measure the abundance of UDGs in 325 spectroscopically-selected groups from the Galaxy And Mass Assembly (GAMA) survey. We make use of the overlapping imaging from the ESO Kilo-Degree Survey (KiDS), from which we can identify galaxies with mean surface brightnesses within their effective radii down to ~25.5 mag arcsec$^{-2}$ in the r-band. We are able to measure a significant overdensity of UDGs (with sizes r_eff > 1.5 kpc) in galaxy groups down to M200=10^12 Msun, a regime where approximately only 1 in 10 groups contains a UDG that we can detect. We combine measurements of the abundance of UDGs in haloes that cover three orders of magnitude in halo mass, finding that their numbers scale quite steeply with halo mass; N_UDG (R<R200) $\propto$ M200^(1.11+/-0.07). To better interpret this, we also measure the mass-richness relation for brighter galaxies down to $M^*_r$+2.5 in the same GAMA groups, and find a much shallower relation of N_Bright (R<R200) $\propto$ M200^(0.78+/-0.05). This shows that UDGs are relatively more abundant, compared to bright galaxies, in massive clusters than in groups. We discuss implications, but whether this difference is related to a higher destruction rate of UDGs in groups, or whether massive haloes have a positive effect on their formation, is not yet clear.
• Large Synoptic Survey Telescope Galaxies Science Roadmap(1708.01617)

The Large Synoptic Survey Telescope (LSST) will enable revolutionary studies of galaxies, dark matter, and black holes over cosmic time. The LSST Galaxies Science Collaboration has identified a host of preparatory research tasks required to leverage fully the LSST dataset for extragalactic science beyond the study of dark energy. This Galaxies Science Roadmap provides a brief introduction to critical extragalactic science to be conducted ahead of LSST operations, and a detailed list of preparatory science tasks including the motivation, activities, and deliverables associated with each. The Galaxies Science Roadmap will serve as a guiding document for researchers interested in conducting extragalactic science in anticipation of the forthcoming LSST era.
• The SAMI Galaxy Survey: mass as the driver of the kinematic morphology - density relation in clusters(1704.01169)

June 15, 2017 astro-ph.GA
We examine the kinematic morphology of early-type galaxies (ETGs) in eight galaxy clusters in the Sydney-AAO Multi-object Integral field spectrograph (SAMI) Galaxy Survey. The clusters cover a mass range of 14.2<log(M_200/M_odot)<15.2 and we measure spatially-resolved stellar kinematics for 315 member galaxies with stellar masses 10.0<log(M_*/M_odot)<11.7 within 1R_200 of the cluster centers. We calculate the spin parameter, lambda_R and use that to classify the kinematic morphology of the galaxies as fast or slow rotators. The total fraction of slow rotators in the early-type galaxy population, F_SR=0.14+/-0.02 and does not depend on host cluster mass. Across the eight clusters, the fraction of slow rotators increases with increasing local overdensity. We also find that the slow-rotator fraction increases at small clustercentric radii (R_cl<0.3R_200), and note that there is also an increase in slow-rotator fraction at R_cl~0.6R_200. The slow rotators at these larger radii reside in cluster substructure. We find the strongest increase in slow-rotator fraction occurs with increasing stellar mass. After accounting for the strong correlation with stellar mass, we find no significant relationship between spin parameter and local overdensity in the cluster environment. We conclude that the primary driver for the kinematic morphology--density relationship in galaxy clusters is the changing distribution of galaxy stellar mass with local environment. The presence of slow rotators in substructure suggests that the cluster kinematic morphology--density relationship is a result of mass segregation of slow-rotating galaxies forming in groups that later merge with clusters and sink to the cluster center via dynamical friction.
• Galaxy and Mass Assembly (GAMA): halo formation times and halo assembly bias on the cosmic web(1612.08595)

June 12, 2017 astro-ph.CO, astro-ph.GA
We present evidence for halo assembly bias as a function of geometric environment. By classifying GAMA galaxy groups as residing in voids, sheets, filaments or knots using a tidal tensor method, we find that low-mass haloes that reside in knots are older than haloes of the same mass that reside in voids. This result provides direct support to theories that link strong halo tidal interactions with halo assembly times. The trend with geometric environment is reversed at large halo mass, with haloes in knots being younger than haloes of the same mass in voids. We find a clear signal of halo downsizing - more massive haloes host galaxies that assembled their stars earlier. This overall trend holds independently of geometric environment. We support our analysis with an in-depth exploration of the L-Galaxies semi-analytic model, used here to correlate several galaxy properties with three different definitions of halo formation time. We find a complex relationship between halo formation time and galaxy properties, with significant scatter. We confirm that stellar mass to halo mass ratio, specific star-formation rate and mass-weighed age are reasonable proxies of halo formation time, especially at low halo masses. Instantaneous star-formation rate is a poor indicator at all halo masses. Using the same semi-analytic model, we create mock spectral observations using complex star-formation and chemical enrichment histories, that approximately mimic GAMA's typical signal-to-noise and wavelength range. We use these mocks to assert how well potential proxies of halo formation time may be recovered from GAMA-like spectroscopic data.
• Galaxy And Mass Assembly: search for a population of high-entropy galaxy groups(1706.03361)

June 11, 2017 astro-ph.GA
Observations with the Chandra X-ray Observatory are used to examine the hot gas properties within a sample of 10 galaxy groups selected from the Galaxy And Mass Assembly survey's optical Friends-of-Friends group catalogue. Our groups have been screened to eliminate spurious and unrelaxed systems, and the effectiveness of this procedure is demonstrated by the detection of intergalactic hot gas in 80 per cent of our sample. However, we find that 9 of the 10 are X-ray underluminous by a mean factor of $\sim$4 compared to typical X-ray-selected samples. Consistent with this, the majority of our groups have gas fractions that are lower and gas entropies somewhat higher than those seen in typical X-ray-selected samples. Two groups, which have high 2{\sigma} lower limits on their gas entropy, are candidates for the population of high-entropy groups predicted by some active galactic nucleus feedback models.
• Galactic googly: the rotation-metallicity bias in the inner stellar halo of the Milky Way(1706.02301)

June 7, 2017 astro-ph.CO, astro-ph.GA
The first and second moments of stellar velocities encode important information about the formation history of the Galactic halo. However, due to the lack of tangential motion and inaccurate distances of the halo stars, the velocity moments in the Galactic halo have largely remained "known unknowns". Fortunately, our off-centric position within the Galaxy allows us to estimate these moments in the galacto-centric frame using the observed radial velocities of the stars alone. We use these velocities coupled with the Hierarchical Bayesian scheme, which allows easy marginalisation over the missing data (the proper-motion, and uncertainty-free distance and line-of-sight velocity), to measure the velocity dispersions, orbital anisotropy ($\beta$) and streaming motion ($v_{\rm rot}$) of the halo main-sequence turn-off (MSTO) and K-giant (KG) stars in the inner stellar halo (r $\lesssim 15$ kpc). We study the metallicity bias in kinematics of the halo stars and observe that the comparatively metal-rich ([Fe/H]$>-1.4$) and the metal-poor ([Fe/H]$\leq - 1.4$) MSTO samples show a clear systematic difference in $v_{\rm rot} \sim 20-40$ km s$^{-1}$, depending on how restrictive the spatial cuts to cull the disk contamination are. The bias is also detected in KG samples but with less certainty. Both MSTO and KG populations suggest that the inner stellar halo of the Galaxy is radially biased i.e. $\sigma_r>\sigma_\theta$ or $\sigma_\phi$ and $\beta \simeq 0.5$. The apparent metallicity contrariety in the rotation velocity among the halo sub-populations supports the co-existence of multiple populations in the galactic halo that may have formed through distinct formation scenarios, i.e. in-situ versus accretion.
• Galaxy And Mass Assembly: the evolution of the cosmic spectral energy distribution from z = 1 to z = 0(1705.07596)

May 22, 2017 astro-ph.GA
We present the evolution of the Cosmic Spectral Energy Distribution (CSED) from $z = 1 - 0$. Our CSEDs originate from stacking individual spectral energy distribution fits based on panchromatic photometry from the Galaxy and Mass Assembly (GAMA) and COSMOS datasets in ten redshift intervals with completeness corrections applied. Below $z = 0.45$, we have credible SED fits from 100 nm to 1 mm. Due to the relatively low sensitivity of the far-infrared data, our far-infrared CSEDs contain a mix of predicted and measured fluxes above $z = 0.45$. Our results include appropriate errors to highlight the impact of these corrections. We show that the bolometric energy output of the Universe has declined by a factor of roughly four -- from $5.1 \pm 1.0$ at $z \sim 1$ to $1.3 \pm 0.3 \times 10^{35}~h_{70}$~W~Mpc$^{-3}$ at the current epoch. We show that this decrease is robust to cosmic variance, SED modelling and other various types of error. Our CSEDs are also consistent with an increase in the mean age of stellar populations. We also show that dust attenuation has decreased over the same period, with the photon escape fraction at 150~nm increasing from $16 \pm 3$ at $z \sim 1$ to $24 \pm 5$ per cent at the current epoch, equivalent to a decrease in $A_\mathrm{FUV}$ of 0.4~mag. Our CSEDs account for $68 \pm 12$ and $61 \pm 13$ per cent of the cosmic optical and infrared backgrounds respectively as defined from integrated galaxy counts and are consistent with previous estimates of the cosmic infrared background with redshift.
• The stellar-to-halo mass relation of GAMA galaxies from 100 square degrees of KiDS weak lensing data(1601.06791)

April 21, 2017 astro-ph.GA
We study the stellar-to-halo mass relation of central galaxies in the range 9.7<log_10(M_*/h^-2 M_sun)<11.7 and z<0.4, obtained from a combined analysis of the Kilo Degree Survey (KiDS) and the Galaxy And Mass Assembly (GAMA) survey. We use ~100 deg^2 of KiDS data to study the lensing signal around galaxies for which spectroscopic redshifts and stellar masses were determined by GAMA. We show that lensing alone results in poor constraints on the stellar-to-halo mass relation due to a degeneracy between the satellite fraction and the halo mass, which is lifted when we simultaneously fit the stellar mass function. At M_sun>5x10^10 h^-2 M_sun, the stellar mass increases with halo mass as ~M_h^0.25. The ratio of dark matter to stellar mass has a minimum at a halo mass of 8x10^11 h^-1 M_sun with a value of M_h/M_*=56_-10^+16 [h]. We also use the GAMA group catalogue to select centrals and satellites in groups with five or more members, which trace regions in space where the local matter density is higher than average, and determine for the first time the stellar-to-halo mass relation in these denser environments. We find no significant differences compared to the relation from the full sample, which suggests that the stellar-to-halo mass relation does not vary strongly with local density. Furthermore, we find that the stellar-to-halo mass relation of central galaxies can also be obtained by modelling the lensing signal and stellar mass function of satellite galaxies only, which shows that the assumptions to model the satellite contribution in the halo model do not significantly bias the stellar-to-halo mass relation. Finally, we show that the combination of weak lensing with the stellar mass function can be used to test the purity of group catalogues.
• Galaxy and Mass Assembly (GAMA): Formation and Growth of Elliptical Galaxies in the Group Environment(1702.07641)

Feb. 24, 2017 astro-ph.GA
There are many proposed mechanisms driving the morphological transformation of disk galaxies to elliptical galaxies. In this paper, we determine if the observed transformation in low mass groups can be explained by the merger histories of galaxies. We measured the group mass-morphology relation for groups from the Galaxy and Mass Assembly group catalogue with masses from 10$^{11}$ - 10$^{15}$ M$_{\odot}$. Contrary to previous studies, the fraction of elliptical galaxies in our more complete group sample increases significantly with group mass across the full range of group mass. The elliptical fraction increases at a rate of 0.163$\pm$0.012 per dex of group mass for groups more massive than 10$^{12.5}$ M$_{\odot}$. If we allow for uncertainties in the observed group masses, our results are consistent with a continuous increase in elliptical fraction from group masses as low as 10$^{11}$M$_{\odot}$. We tested if this observed relation is consistent with merger activity using a GADGET-2 dark matter simulation of the galaxy groups. We specified that a simulated galaxy would be transformed to an elliptical morphology either if it experienced a major merger or if its cumulative mass gained from minor mergers exceeded 30 per cent of its final mass. We then calculated a group mass-morphology relation for the simulations. The position and slope of the simulated relation were consistent with the observational relation, with a gradient of 0.184$\pm$0.010 per dex of group mass. These results demonstrate a strong correlation between the frequency of merger events and disk-to-elliptical galaxy transformation in galaxy group environments.
• Galaxy And Mass Assembly (GAMA): the Stellar Mass Budget of Galaxy Spheroids and Disks(1608.05526)

Aug. 19, 2016 astro-ph.GA
We build on a recent photometric decomposition analysis of 7506 Galaxy and Mass Assembly (GAMA) survey galaxies to derive stellar mass function fits to individual spheroid and disk component populations down to a lower mass limit of log(M_*/M_sun)= 8. We find that the spheroid/disk mass distributions for individual galaxy morphological types are well described by single Schechter function forms. We derive estimates of the total stellar mass densities in spheroids (rho_spheroid = 1.24+/-0.49 * 10^8 M_sun Mpc^-3 h_0.7) and disks (rho_disk = 1.20+/-0.45 * 10^8 M_sun Mpc^-3 h_0.7), which translates to approximately 50% of the local stellar mass density in spheroids and 48% in disks. The remaining stellar mass is found in the dwarf "little blue spheroid" class, which is not obviously similar in structure to either classical spheroid or disk populations. We also examine the variation of component mass ratios across galaxy mass and group halo mass regimes, finding the transition from spheroid to disk mass dominance occurs near galaxy stellar mass ~10^11 M_sun and group halo mass ~10^12.5 M_sun/h. We further quantify the variation in spheroid-to-total mass ratio with group halo mass for central and satellite populations as well as the radial variation of this ratio within groups.
• Galaxy And Mass Assembly (GAMA): $\mathcal{M_\star}-R_{\rm e}$ relations of $z=0$ bulges, discs and spheroids(1607.01096)

July 5, 2016 astro-ph.CO, astro-ph.GA
We perform automated bulge + disc decomposition on a sample of $\sim$7500 galaxies from the Galaxy And Mass Assembly (GAMA) survey in the redshift range of 0.002$<$z$<$0.06 using SIGMA, a wrapper around GALFIT3. To achieve robust profile measurements we use a novel approach of repeatedly fitting the galaxies, varying the input parameters to sample a large fraction of the input parameter space. Using this method we reduce the catastrophic failure rate significantly and verify the confidence in the fit independently of $\chi^2$. Additionally, using the median of the final fitting values and the 16$^{th}$ and 84$^{th}$ percentile produces more realistic error estimates than those provided by GALFIT, which are known to be underestimated. We use the results of our decompositions to analyse the stellar mass - half-light radius relations of bulges, discs and spheroids. We further investigate the association of components with a parent disc or elliptical relation to provide definite $z=0$ disc and spheroid $\mathcal{M_\star}-R_{\rm e}$ relations. We conclude by comparing our local disc and spheroid $\mathcal{M_\star}-R_{\rm e}$ to simulated data from EAGLE and high redshift data from CANDELS-UDS. We show the potential of using the mass-size relation to study galaxy evolution in both cases but caution that for a fair comparison all data sets need to be processed and analysed in the same manner.
• Science Impacts of the SPHEREx All-Sky Optical to Near-Infrared Spectral Survey: Report of a Community Workshop Examining Extragalactic, Galactic, Stellar and Planetary Science(1606.07039)

June 22, 2016 astro-ph.CO, astro-ph.GA
SPHEREx is a proposed SMEX mission selected for Phase A. SPHEREx will carry out the first all-sky spectral survey and provide for every 6.2" pixel a spectra between 0.75 and 4.18 $\mu$m [with R$\sim$41.4] and 4.18 and 5.00 $\mu$m [with R$\sim$135]. The SPHEREx team has proposed three specific science investigations to be carried out with this unique data set: cosmic inflation, interstellar and circumstellar ices, and the extra-galactic background light. It is readily apparent, however, that many other questions in astrophysics and planetary sciences could be addressed with the SPHEREx data. The SPHEREx team convened a community workshop in February 2016, with the intent of enlisting the aid of a larger group of scientists in defining these questions. This paper summarizes the rich and varied menu of investigations that was laid out. It includes studies of the composition of main belt and Trojan/Greek asteroids; mapping the zodiacal light with unprecedented spatial and spectral resolution; identifying and studying very low-metallicity stars; improving stellar parameters in order to better characterize transiting exoplanets; studying aliphatic and aromatic carbon-bearing molecules in the interstellar medium; mapping star formation rates in nearby galaxies; determining the redshift of clusters of galaxies; identifying high redshift quasars over the full sky; and providing a NIR spectrum for most eROSITA X-ray sources. All of these investigations, and others not listed here, can be carried out with the nominal all-sky spectra to be produced by SPHEREx. In addition, the workshop defined enhanced data products and user tools which would facilitate some of these scientific studies. Finally, the workshop noted the high degrees of synergy between SPHEREx and a number of other current or forthcoming programs, including JWST, WFIRST, Euclid, GAIA, K2/Kepler, TESS, eROSITA and LSST.
• Extra-galactic background light measurements from the far-UV to the far-IR from deep ground and space-based galaxy counts(1605.01523)

May 5, 2016 astro-ph.CO, astro-ph.GA
We combine wide and deep galaxy number-count data from GAMA, COSMOS/G10, HST ERS, HST UVUDF and various near-, mid- and far- IR datasets from ESO, Spitzer and Herschel. The combined data range from the far-UV (0.15microns) to far-IR (500microns), and in all cases the contribution to the integrated galaxy light (IGL) of successively fainter galaxies converges. Using a simple spline fit, we derive the IGL and the extrapolated-IGL in all bands. We argue undetected low surface brightness galaxies and intra-cluster/group light is modest, and that our extrapolated-IGL measurements are an accurate representation of the extra-galactic background light. Our data agree with most earlier IGL estimates and with direct measurements in the far-IR, but disagree strongly with direct estimates in the optical. Close agreement between our results and recent very high-energy experiments (H.E.S.S. and MAGIC), suggest that there may be an additional foreground affecting the direct estimates. The most likely culprit could be the adopted Zodiacal light model. Finally we use a modified version of the two-component model to integrate the EBL and obtain measurements of the Cosmic Optical Background (COB) and Cosmic Infrared Background (CIB) of: $24^{+4}_{-4}$nWm$^{-2}$sr$^{-1}$ and $26^{+5}_{-5}$nWm$^{-2}$sr$^{-1}$ respectively (48:52%). Over the next decade, upcoming space missions such as Euclid and WFIRST, have the capacity to reduce the COB error to $<1\%$, at which point comparisons to the very high energy data could have the potential to provide a direct detection and measurement of the reionisation field.
• Dependence of GAMA galaxy halo masses on the cosmic web environment from 100 square degrees of KiDS weak lensing data(1604.07233)

April 25, 2016 astro-ph.GA
Galaxies and their dark matter haloes are part of a complex network of mass structures, collectively called the cosmic web. Using the tidal tensor prescription these structures can be classified into four cosmic environments: voids, sheets, filaments and knots. As the cosmic web may influence the formation and evolution of dark matter haloes and the galaxies they host, we aim to study the effect of these cosmic environments on the average mass of galactic haloes. To this end we measure the galaxy-galaxy lensing profile of 91,195 galaxies, within 0.039 < z < 0.263, from the spectroscopic Galaxy And Mass Assembly (GAMA) survey, using ~100 square degrees of overlapping data from the Kilo-Degree Survey (KiDS). In each of the four cosmic environments we model the contributions from group centrals, satellites and neighbouring groups to the stacked galaxy-galaxy lensing profiles. After correcting the lens samples for differences in the stellar mass distribution, we find no dependence of the average halo mass of central galaxies on their cosmic environment. We do find a significant increase in the average contribution of neighbouring groups to the lensing profile in increasingly dense cosmic environments. We show, however, that the observed effect can be entirely attributed to the galaxy density at much smaller scales (within 4 Mpc/h), which is correlated with the density of the cosmic environments. Within our current uncertainties we find no direct dependence of galaxy halo mass on their cosmic environment.
• Galaxy And Mass Assembly (GAMA): Stellar mass growth of spiral galaxies in the cosmic web(1601.03391)

Jan. 13, 2016 astro-ph.GA
We look for correlated changes in stellar mass and star formation rate along filaments in the cosmic web by examining the stellar masses and UV-derived star formation rates (SFR) of 1,799 ungrouped and unpaired spiral galaxies that reside in filaments. We devise multiple distance metrics to characterise the complex geometry of filaments, and find that galaxies closer to the cylindrical centre of a filament have higher stellar masses than their counterparts near the periphery of filaments, on the edges of voids. In addition, these peripheral spiral galaxies have higher specific star formation rates (SSFR) at a given mass. Complementing our sample of filament spiral galaxies with spiral galaxies in tendrils and voids, we find that the average SFR of these objects in different large scale environments are similar to each other with the primary discriminant in SFR being stellar mass, in line with previous works. However, the distributions of SFRs are found to vary with large-scale environment. Our results thus suggest a model in which in addition to stellar mass as the primary discriminant, the large-scale environment is imprinted in the SFR as a second order effect. Furthermore, our detailed results for filament galaxies suggest a model in which gas accretion from voids onto filaments is primarily in an orthogonal direction. Overall, we find our results to be in line with theoretical expectations of the thermodynamic properties of the intergalactic medium in different large-scale environments.
• Galaxy And Mass Assembly (GAMA): the Stellar Mass Budget by Galaxy Type(1512.02342)

Dec. 8, 2015 astro-ph.GA
We report an expanded sample of visual morphological classifications from the Galaxy and Mass Assembly (GAMA) survey phase two, which now includes 7,556 objects (previously 3,727 in phase one). We define a local (z <0.06) sample and classify galaxies into E, S0-Sa, SB0-SBa, Sab-Scd, SBab-SBcd, Sd-Irr, and "little blue spheroid" types. Using these updated classifications, we derive stellar mass function fits to individual galaxy populations divided both by morphological class and more general spheroid- or disk-dominated categories with a lower mass limit of log(Mstar/Msun) = 8 (one dex below earlier morphological mass function determinations). We find that all individual morphological classes and the combined spheroid-/bulge-dominated classes are well described by single Schechter stellar mass function forms. We find that the total stellar mass densities for individual galaxy populations and for the entire galaxy population are bounded within our stellar mass limits and derive an estimated total stellar mass density of rho_star = 2.5 x 10^8 Msun Mpc^-3 h_0.7, which corresponds to an approximately 4% fraction of baryons found in stars. The mass contributions to this total stellar mass density by galaxies that are dominated by spheroidal components (E and S0-Sa classes) and by disk components (Sab-Scd and Sd-Irr classes) are approximately 70% and 30%, respectively.
• Galaxy And Mass Assembly (GAMA): Panchromatic Data Release (far-UV --- far-IR) and the low-z energy budget(1508.02076)

Nov. 20, 2015 astro-ph.CO, astro-ph.GA
We present the GAMA Panchromatic Data Release (PDR) constituting over 230deg$^2$ of imaging with photometry in 21 bands extending from the far-UV to the far-IR. These data complement our spectroscopic campaign of over 300k galaxies, and are compiled from observations with a variety of facilities including: GALEX, SDSS, VISTA, WISE, and Herschel, with the GAMA regions currently being surveyed by VST and scheduled for observations by ASKAP. These data are processed to a common astrometric solution, from which photometry is derived for 221,373 galaxies with r<19.8 mag. Online tools are provided to access and download data cutouts, or the full mosaics of the GAMA regions in each band. We focus, in particular, on the reduction and analysis of the VISTA VIKING data, and compare to earlier datasets (i.e., 2MASS and UKIDSS) before combining the data and examining its integrity. Having derived the 21-band photometric catalogue we proceed to fit the data using the energy balance code MAGPHYS. These measurements are then used to obtain the first fully empirical measurement of the 0.1-500$\mu$m energy output of the Universe. Exploring the Cosmic Spectral Energy Distribution (CSED) across three time-intervals (0.3-1.1Gyr, 1.1-1.8~Gyr and 1.8---2.4~Gyr), we find that the Universe is currently generating $(1.5 \pm 0.3) \times 10^{35}$ h$_{70}$ W Mpc$^{-3}$, down from $(2.5 \pm 0.2) \times 10^{35}$ h$_{70}$ W Mpc$^{-3}$ 2.3~Gyr ago. More importantly, we identify significant and smooth evolution in the integrated photon escape fraction at all wavelengths, with the UV escape fraction increasing from 27(18)% at z=0.18 in NUV(FUV) to 34(23)% at z=0.06. The GAMA PDR will allow for detailed studies of the energy production and outputs of individual systems, sub-populations, and representative galaxy samples at $z<0.5$. The GAMA PDR can be found at: http://gama-psi.icrar.org/
• Two-phase galaxy evolution: the cosmic star-formation histories of spheroids and discs(1301.0979)

Nov. 20, 2015 astro-ph.CO
From two very simple axioms: (1) that AGN activity traces spheroid formation, and (2) that the cosmic star-formation history is dominated by spheroid formation at high redshift, we derive simple expressions for the star-formation histories of spheroids and discs, and their implied metal enrichment histories. Adopting a Baldry-Glazebrook initial mass function we use these relations and apply PEGASE.2 to predict the z=0 cosmic spectral energy distributions (CSEDs) of spheroids and discs. The model predictions compare favourably to the dust-corrected CSED recently reported by the Galaxy And Mass Assembly (GAMA) team from the FUV through to the K band. The model also provides a reasonable fit to the total stellar mass contained within spheroid and disc structures as recently reported by the Millennium Galaxy Catalogue team. Three interesting inferences can be made following our axioms: (1) there is a transition redshift at z ~ 1.7 at which point the Universe switches from what we refer to as "hot mode evolution" (i.e., spheroid formation/growth via mergers and/or collapse) to what we term "cold mode evolution" (i.e., disc formation/growth via gas infall and minor mergers); (2) there is little or no need for any pre-enrichment prior to the main phase of star-formation; (3) in the present Universe mass-loss is fairly evenly balanced with star-formation holding the integrated stellar mass density close to a constant value. The model provides a simple prediction of the energy output from spheroid and disc projenitors, the build-up of spheroid and disc mass, and the mean metallicity enrichment of the Universe.


text:
## CryptoDB

### Paper: On the Exact Round Complexity of Secure Three-Party Computation

Authors: Arpita Patra Divya Ravi DOI: 10.1007/s00145-021-09404-0 Search ePrint Search Google We settle the exact round complexity of three-party computation (3PC) in honest-majority setting, for a range of security notions such as selective abort, unanimous abort, fairness and guaranteed output delivery. It is a folklore that the implication holds from the guaranteed output delivery to fairness to unanimous abort to selective abort. We focus on computational security and consider two network settings—pairwise-private channels without and with a broadcast channel. In the minimal setting of pairwise-private channels, 3PC with selective abort is known to be feasible in just two rounds, while guaranteed output delivery is infeasible to achieve irrespective of the number of rounds. Settling the quest for exact round complexity of 3PC in this setting, we show that three rounds are necessary and sufficient for unanimous abort and fairness. Extending our study to the setting with an additional broadcast channel, we show that while unanimous abort is achievable in just two rounds, three rounds are necessary and sufficient for fairness and guaranteed output delivery. Our lower bound results extend for any number of parties in honest majority setting and imply tightness of several known constructions. While our lower bounds extend to the common reference string (CRS) model, all our upper bounds are in the plain model. The fundamental concept of garbled circuits underlies all our upper bounds. Concretely, our constructions involve transmitting and evaluating only constant number of garbled circuits. Assumption-wise, our constructions rely on injective (one-to-one) one-way functions.
##### BibTeX
@article{jofc-2021-31753,
title={On the Exact Round Complexity of Secure Three-Party Computation},
journal={Journal of Cryptology},
publisher={Springer},
volume={34},
doi={10.1007/s00145-021-09404-0},
author={Arpita Patra and Divya Ravi},
year=2021
}

format: plain


text:
# Thin Plate Spline interpolation of scattered $z(x,y)$ data

I am trying to understand Thin Plate Spline interpolation of scattered data. As I understand it TPS is just a special case of Radial Basis Function interpolation:

$$z(x,y) = p(x,y) + \sum_i l_i\phi(r)$$ where p(x,y) is a polynomial and $\ \phi$ is a RBF.

In the case of TPS: $\ \phi = r^2\ln(r)$

Should not the RBF decrease with r? In the case of inverse distance weighted interpolation the weight of each point goes from +infinity at r = 0 to 0 at r = + infinity.

Which order of polynomial should I use? I understand that p(x,y) should compensate for offset and maybe a linear trend, so that would indicate: p(x,y) = ax+by+c. But why stop there?

The TPS is required to go exactly trough each point: $$z_i = p(x_i,y_i) + \sum_j l_j\phi(r_i)$$

This can be written as a matrix equation: $$[\Phi, X, Y, 1]*[L, a, b, c]'=Z$$ where $\ \Phi$ is a nxn matrix, X, Y, L and Z are nx1 vectors.

This is underdetermined: there are n + 3 unknowns and n equations. Therefore one have to impose some additional requirements. One requires: $$\sum_i x_i*l_i = 0$$ $$\sum_i y_i*l_i = 0$$ $$\sum_i 1*l_i = 0$$ and calls this orthogonality. But what does this mean? I am used to orthogonal function expansions and understand that $\ <x,b(x,y)> = 0$ for some basis function,b, would mean that x and b(x,y) are orthogonal and that would be a good thing since it makes it easy to find the coefficients of b by simply taking the inner product of b and the function to be expanded. But in this context? And why is the coefficents of the RBFs that are in the orthogonality requirement and not the RBFs themselves?

However this means that we now have n equations for n unknows. Introducing $\ P=[X,Y,1]$ and $\ C = [a,b,c]'$ our first matrix equation can be written as: $$[\Phi | P][L|C]'=Z$$ and the orthogonality requirements becomes a vertical concatenation to this: $$[P' | 0]*[L | C]' = 0$$

Which method would you recommend to solve this equation?

Which functions does this correspond to in Lapack?

If the number of samples, n, is large this computation becomes heavy. Is there some easy and sensible way to improve upon this? I was considering introducing a cut-off radius and any points further away than this would get the $\ \phi$ = 0 value. This would lead to a sparse matrix equation. However this stumbles on my first question: why does not the RBF decay to 0 as r goes to infinity?

• Excellent question. – nbubis Jul 21 '15 at 13:11

This answer will be by no means complete but I can answer few things.

Should not the RBF decrease with r? In the case of inverse distance weighted interpolation the weight of each point goes from +infinity at r = 0 to 0 at r = + infinity.

No it does not have to. Let's have a look at RBF interpolation in 1D. Take $\phi(x) = |x|$ as your RBF and you get standard linear interpolation.

You can define Lagrange interpolation basis $L_i(x) = \sum_j l_j \phi(x-x_j)$ satisfying $L_i(x_j) = \delta_{ij}$. Than there functions $L_i(x)$ behave how you would expect. They are sort of local i.e. they are almost zero except neighborhood of $x_i$.

Example for $\phi(x)=|x|$ and points $x_0=0,x_1=1,x_2=2$. Than:

$$L_0(x) = -0.25|x|+0.5|x-1| +0.25|x-2|$$ $$L_1(x) = 0.5|x|- |x-1|+0.5|x-2|$$ $$L_2(x) = +0.25|x|+0.5|x-1| -0.25|x-2|$$

If the number of samples, n, is large this computation becomes heavy. Is there some easy and sensible way to improve upon this?

Yes there is. If you use thin-plate in 2d than fast multipole method has quite easy form. There is also some moment based method I know nothing about.

With the polynomials I can't help. When I wrote some RBF interpolation I did first least square polynomial interpolation. And than I did RBF interpolation to further improve interpolation.

So first I find polynomial $p$ of given degree which minimalize $\sum_i |p(x_i)-f_i|$. And than I find coefficients $l_i$ that $\sum_i l_i \phi(x_j -x_i) = f_j-p(x_j)$ holds.

For complete information about RBF see this book.

• Thanks Tom! I still find it hard to understand that a RBF can increase with r though. Does this not mean that the value in an interpolated point is more influenced by far away points than near points? Sounds like this should be a recepie for discontinous surfaces. – Andy Jun 8 '13 at 15:09
• Doing LSE first to estimate the global trend polynomial followed by RBF sounds like a convenient way to do things since it results in a simpler matrix equation for the RBF. Thanks! – Andy Jun 8 '13 at 15:11


text:
# 5 Spending our data

There are several steps to create a useful model, including parameter estimation, model selection and tuning, and performance assessment. At the start of a new project, there is usually an initial finite pool of data available for all these tasks. How should the data be applied to these steps? The idea of data spending is an important first consideration when modeling, especially as it relates to empirical validation.

When there are copious amounts of data available, a smart strategy is to allocate specific subsets of data for different tasks, as opposed to allocating the largest possible amount to the model parameter estimation only. There may be questions about many modeling project steps that must be answered with limited prior knowledge. For example, one possible strategy (when both data and predictors are abundant) is to spend a specific subset of data to determine which predictors are informative, before considering parameter estimation at all.

As data are reused for multiple tasks, certain risks increase, such as the risks of adding bias or large effects from methodological errors.

If the initial pool of data available is not huge, there will be some overlap of how and when our data is “spent” or allocated, and a solid methodology for data spending is important. This chapter demonstrates the basics of splitting our initial pool of samples for different purposes.

## 5.1 Common methods for splitting data

The primary approach for empirical model validation is to split the existing pool of data into two distinct sets. Some observations are used to develop and optimize the model. This training set is usually the majority of the data. These data are a sandbox for model building where different models can be fit, feature engineering strategies are investigated, and so on. We as modeling practitioners spend the vast majority of the modeling process using the training set as the substrate to develop the model.

The other portion of the observations are placed into the test set. This is held in reserve until one or two models are chosen as the methods that are most likely to succeed. The test set is then used as the final arbiter to determine the efficacy of the model. It is critical to only look at the test set once; otherwise, it becomes part of the modeling process.

How should we conduct this split of the data? This depends on the context.

Suppose we allocate 80% of the data to the training set and the remaining 20% for testing. The most common method is to use simple random sampling. The rsample package has tools for making data splits such as this; the function initial_split() was created for this purpose. It takes the data frame as an argument as well as the proportion to be placed into training. Using the previous data frame produced by the code snippet from the summary in Section 4.2:

library(tidymodels)
tidymodels_prefer()

# Set the random number stream using set.seed() so that the results can be
# reproduced later.
set.seed(123)

# Save the split information for an 80/20 split of the data
ames_split <- initial_split(ames, prop = 0.80)
ames_split
#> <Analysis/Assess/Total>
#> <2344/586/2930>

The printed information denotes the amount of data in the training set ($$n = 2,344$$), the amount in the test set ($$n = 586$$), and the size of the original pool of samples ($$n = 2,930$$).

The object ames_split is an rsplit object and only contains the partitioning information; to get the resulting data sets, we apply two more functions:

ames_train <- training(ames_split)
ames_test  <-  testing(ames_split)

dim(ames_train)
#> [1] 2344   74

These objects are data frames with the same columns as the original data but only the appropriate rows for each set.

Simple random sampling is appropriate in many cases but there are exceptions. When there is a dramatic class imbalance in classification problems, one class occurs much less frequently than another. Using a simple random sample may haphazardly allocate these infrequent samples disproportionately into the training or test set. To avoid this, stratified sampling can be used. The training/test split is conducted separately within each class and then these subsamples are combined into the overall training and test set. For regression problems, the outcome data can be artificially binned into quartiles and then stratified sampling conducted four separate times. This is an effective method for keeping the distributions of the outcome similar between the training and test set.

The distribution of the sale price outcome for the Ames housing data is shown in Figure 5.1. As previously discussed, the sale price distribution is right-skewed, with proportionally more inexpensive houses than expensive houses on either side of the center of the distribution. The worry here is that the more expensive houses would not be represented in the training set well with simple splitting; this would increase the risk that our model would be ineffective at predicting the price for such properties. The dotted vertical lines in Figure 5.1 indicate the four quartiles for these data. A stratified random sample would conduct the 80/20 split within each of these data subsets and then pool the results together. In rsample, this is achieved using the strata argument:

set.seed(123)
ames_split <- initial_split(ames, prop = 0.80, strata = Sale_Price)
ames_train <- training(ames_split)
ames_test  <-  testing(ames_split)

dim(ames_train)
#> [1] 2342   74

Only a single column can be used for stratification.

There is very little downside to using stratified sampling.

Are there situations when random sampling is not the best choice? One case is when the data have a significant time component, such as time series data. Here, it is more common to use the most recent data as the test set. The rsample package contains a function called initial_time_split() that is very similar to initial_split(). Instead of using random sampling, the prop argument denotes what proportion of the first part of the data should be used as the training set; the function assumes that the data have been pre-sorted in an appropriate order.

## 5.2 What proportion should be used?

The proportion of data that should be allocated for splitting is highly dependent on the context of the problem at hand. Too little data in the training set hampers the model’s ability to find appropriate parameter estimates. Conversely, too little data in the test set lowers the quality of the performance estimates. There are parts of the statistics community that eschew test sets in general because they believe all of the data should be used for parameter estimation. While there is merit to this argument, it is good modeling practice to have an unbiased set of observations as the final arbiter of model quality. A test set should be avoided only when the data are pathologically small.

## 5.3 What about a validation set?

Previously, when describing the goals of data splitting, we singled out the test set as the data that should be used to conduct a proper evaluation of model performance on the final model(s). This begs the question of, “How can we tell what is best if we don’t measure performance until the test set?”

It is common to hear about validation sets as an answer to this question, especially in the neural network and deep learning literature. The validation set was originally defined in the early days of neural networks when researchers realized that measuring performance by re-predicting the training set samples led to results that were overly optimistic (significantly, unrealistically so). This led to models that overfit, meaning that they performed very well on the training set but poorly on the test set9. To combat this issue, a small validation set of data were held back and used to measure performance as the network was trained. Once the validation set error rate began to rise, the training would be halted. In other words, the validation set was a means to get a rough sense of how well the model performed prior to the test set.

It is largely semantics as to whether validation sets are a subset of the training set or a third allocation in the initial split of the data.

Validation sets are discussed more in Section 10.2.2 as a special case of resampling methods that are used on the training set.

## 5.4 Multi-level data

With the Ames housing data, a property is considered to be the independent experimental unit. It is safe to assume that, statistically, the data from a property are independent. For other applications, that is not always the case:

• For longitudinal data, the same independent experimental unit can be measured over multiple time points. An example would be a human subject in a medical trial.

• A batch of manufactured product might also be considered the independent experimental unit. In repeated measures designs, replicate data points from a batch are collected.

• Johnson et al. (2018) report an experiment where different trees were sampled across the top and bottom portions of a stem. Here, the tree is the experimental unit and the data hierarchy is sample within stem position within tree.

Chapter 9 of Kuhn and Johnson (2020) contains other examples.

In these situations, the data set will have multiple rows per experimental unit. Simple resampling across rows would lead to some data within an experimental unit being in the training set and others in the test set. Data splitting should occur at the independent experimental unit level of the data. For example, to produce an 80/20 split of the data, 80% of the experimental units should be allocated for the training set.

## 5.5 Other considerations

Throughout this book, notice which data are exposed to the model at any given time. Remember that it is critical to quarantine the test set from any model building activities.

The problem of information leakage occurs when data outside of the training set are used in the modeling process.

For example, in a machine learning competition, the test set data might be provided without the true outcome values so that the model can be scored and ranked. One potential method for improving the score might be to fit the model using the training set points that are most similar to the test set values. While the test set isn’t directly used to fit the model, it still has a heavy influence. In general, this technique is highly problematic since it reduces the generalization error of the model to optimize performance on a specific data set. There are more subtle ways that the test set data can be utilized during training. Keeping the training data in a separate data frame from the test set is one small check to make sure that information leakage does not occur by accident.

In later chapters we discuss techniques to subsample the training set to mitigate specific issues (e.g., class imbalances). This is a valid and common technique that deliberately results in the training set data diverging from the population from which the data were drawn. It is critical that the test set continue to mirror what the model would encounter in the wild. In other words, the test set should always resemble new data that will be given to the model.

Finally, at the beginning of this chapter, we warned about using the same data for different tasks. Chapter 10 will discuss solid, data-driven methodologies for data usage that will reduce the risks related to bias, overfitting, and other issues. Many of these methods mirror the data-splitting tools shown in this chapter.

## 5.6 Chapter summary

Data splitting is the fundamental tool for empirical validation of models. Even in the era of unrestrained data collection, a typical modeling project has a limited amount of appropriate data and wise “spending” of a project’s data is necessary. In this chapter, we discussed several strategies for partitioning the data into distinct groups for modeling and evaluation.

At this checkpoint, the important code snippets are:

library(tidymodels)
data(ames)
ames <- ames %>% mutate(Sale_Price = log10(Sale_Price))

set.seed(123)
ames_split <- initial_split(ames, prop = 0.80, strata = Sale_Price)
ames_train <- training(ames_split)
ames_test  <-  testing(ames_split)

### REFERENCES

Johnson, D, P Eckart, N Alsamadisi, H Noble, C Martin, and R Spicer. 2018. “Polar Auxin Transport Is Implicated in Vessel Differentiation and Spatial Patterning During Secondary Growth in Populus.” American Journal of Botany 105 (2): 186–96.

Kuhn, M, and K Johnson. 2020. Feature Engineering and Selection: A Practical Approach for Predictive Models. CRC Press.

1. This is discussed in much greater detail in Section 12.3.↩


text:
Irrational Numbers

1. Sep 14, 2006

Pythagorean

don't exist!!!

and spiders are your new gods, worship them...

2. Sep 14, 2006

BSMSMSTMSPHD

Um.... okay.

:uhh:

3. Sep 14, 2006

SpaceTiger

Staff Emeritus
New? Where have you been?

4. Sep 14, 2006

chroot

Staff Emeritus
Irrational numbers are just a little nutty. It's the imaginary numbers that don't exist.

- Warre

5. Sep 14, 2006

Pythagorean

Heh, i was trying to keep in character.

(The Pythagoreans tried to hide the existance of irrational numbers in nature since it assaulted their belief that the universe was made completely of integer ratios)

6. Sep 14, 2006

chroot

Staff Emeritus
I'll drown you, you dirty rat!

- Warren

7. Sep 14, 2006

Gokul43201

Staff Emeritus
Q: Why did the Pythagorean cross the road?

8. Sep 14, 2006

franznietzsche

To get to the other side?

9. Sep 14, 2006

Gokul43201

Staff Emeritus
No. It was a trick quetion. Pythagoreans are forbidden from walking on roads! <a little poetic license there>

But if I were guessing along your lines, I'd have gone for: "to get to the opposite side".

Last edited: Sep 14, 2006
10. Sep 14, 2006

chroot

Staff Emeritus
Only diagonal ones.

- Warren

11. Sep 14, 2006

Staff Emeritus
ALL numbers are imaginary! So what?

12. Sep 14, 2006

franznietzsche

No, all numbers are complex. Silly.

13. Sep 14, 2006

Pythagorean

interesting. Do you know the reasoning for this? Is it a theological constraint?

14. Sep 14, 2006

Pythagorean

this is for the generally uninformed public. Absolute geniuses like you and I know the form of perfection when we see it.

http://www.timart.be/Npaginas/foto/wolf_spider.jpg [Broken]

omg! 6:8 is 3:4 is *fap*fap*fap*drool*worship*

Last edited by a moderator: May 2, 2017
15. Sep 14, 2006

Gokul43201

Staff Emeritus
Not looked much into it.You've heard of the other crazy-@ rules they had, right?

http://everything2.com/index.pl?node=Pythagorean

16. Sep 14, 2006

Pythagorean

Oh man, this is going to take way too much energy.....

but atheism is so boring... :/