Penn EA Residency Takeaways
Thanks to Zack Dugue, Oliver Habryka, Adam Krivka, and the Penn EA team for useful comments.
In September 2021, sydv and I (Thomas Kwa) helped revive the University of Pennsylvania EA group through a residency. We think this went very well, and the ~300 hours invested during the 3 weeks we were there probably sped up the group by 4 months, and possibly made the group significantly better overall [3]. As of November 2021, Penn EA is currently a thriving group with six organizers, ~30 Intro Program participants and ~20 weekly dinner attendees, and potential to grow further into its huge 10,000 undergrad population, whereas Iâd guess the counterfactual looks something like slowly building up to this size over the course of ~8 months with significant risk of the group dying again.
What is a residency?
In (our model of) a residency, one or two EA community builders travel to a large university at the start of the school year, and spend at least 1 FTE building a new or existing EA student group. At the end, the group is handed off to students, and there might be a retreat for new organizers. The primary goal is to build organizer capacity to rapidly make the group large and self-sustaining (at least 2 students with 10h/âweek each); a secondary goal is general community-building.
Penn is strong evidence that this model can work well. Itâs only one data point, but we think a lot of the success is generalizable. Note there were other residencies this fall that didnât go as well.
Misc comments
Sydney (a Stanford student) and I (a Caltech student) were able to invest this much time despite being full-time students with our own groups to run because Penn starts 3 weeks earlier than Caltech or Stanford. [1]
Our time was mostly spent building the website, tabling (sitting outside in a high-traffic area and advertising the club), having one-on-ones with prospective organizers and others who signed up through the website, organizing and speaking at events like intro sessions and dinners, other advertising (like flyers) and organizing a retreat (see below). Of these actions, we spent the most time tabling. But one-on-ones and the retreat were the most efficient use of time, as they build organizer capacity. The other actions are still roughly as good as standard university organizer time, given the importance of the first few weeks of uni.
Tabling: At Penn, itâs really hard to send emails to the whole school. Emails are necessary to get attendance at events. We used the labor-intensive but very effective strategy of tabling to collect ~900 emails, and averaged ~20 emails and ~1 Intro Program application per hour (more on club fair days). Just as importantly, we met a couple more potential organizers. We highly recommend that group organizers read our Guide to Tabling, because following these practices more than doubled our tabling effectiveness.
One-on-ones: Sydney did most of these; the goal was to identify potential organizers /â active members and what they might contribute to the club, and also just get to know people. I think there were about 20 one-hour 1-1s.
Retreat: We ran a weekend retreat for Mid-Atlantic region EA group organizers after the residency, with a total of ~30 people including ~8 from Penn. Retreats are out of scope of this post, but we think theyâre great and pair well with residencies; all the newly excited people youâve had 1-1s with can talk to each other and learn more about EA and EA community building.
We did expensive messaging experiments which were inconclusive (we technically started 3 different clubs, and tabled once as Penn Rationality and a couple times as Penn Impactful Careers). In the end we stuck with neutral EA branding, roughly âwe want to help students solve the worldâs most pressing problemsâ. I think we needed to do messaging experiments, because there are probably a lot of campuses where (say) career-first messaging would do 50% better, and itâs hard to read the culture of a campus without trying messaging. But maybe thereâs a less time-consuming way.
While at Penn we met with other uni EA groups in the area, went to EA Philadelphia events with potential organizers, and once drove to Overcoming Bias NYC. We think this was really good for networking.
We think there is room for efficiency improvement: I think I could do this at a similar university in ~25% less time given more advance planning, better prioritization, and lessons learned.
Requirements for a residency
Connections: We already knew Ashley Lin, now co-president of the club, and some other people at Penn. We met Brandon Sayler, the other co-president, in the first week. We knew that both were fairly excited about community building, and Ashley was on a gap year and had ~20 h/âweek to spare. This helped with administrative things (establishing the club as an official Penn student group, which requires Penn students) and handing off responsibility at the end of the residency. To get connections, you can search the EA Hub, but not everyone is there, so you can also contact someone who knows hundreds of EAs worldwide, like Sydney or Kuhan.
Funding: We spent a few thousand dollars on various supplies, including a lot on food for events and smaller amounts on advertising materials. We obtained funding by verbally confirming with CEA about the expected scope and cost of the project beforehand, and submitting a reimbursement request afterwards. It would also have been possible to get funding from the EA Infrastructure fund. Once organizers are found, CBGs are a great way to increase available organizer time.
Lots of time: I think there were increasing returns to the first 100-150 hours of the residency. This is because the first week was mostly spent getting situated, and the basic infrastructure of the club (website, finding organizers) also took a lot of time, and Penn is a large school. I donât know if a residency could succeed with less than 1 FTE (full-time equivalent, 40h/âweek). That might be more of an advisory /â campus specialist role.
The right school and situation: Penn students are on average pretty receptive to (our framing of) EA. Iâm not sure exactly why this is, maybe they have an entrepreneurial mindset, havenât committed to career plans, or arenât too excited by the popular options of consulting and tech jobs. Penn EA used to be pretty big in ~2017 before going dormant due to lack of organizer time, so we knew a group could succeed. We also think a residency at a school with an extant group might be more challenging than restarting a group or building one from scratch.
Prioritization and taking initiative: Thereâs never enough time to do all the advertising /â organizing you want to. But sometimes the best thing to do is >2x as good as the others and requires a lot of agency.
Infrastructure: We knew how to use Wix, Canva, Mailchimp, and Airtable. Also, we picked up best practices for starting EA groups from talking to other organizers from Stanford, many other uni groups, and uni group support people like Emma Abele during a previous retreat. We continued to share residency-specific best practices during the residency with other big groups like MIT, and we expect that had there been even more support and dialogue, the residency could have gone even better.
FAQ
Why do you think Penn EA was sped up by ~4 months?
There are a bunch of caveats; see [3]. Sydney thinks it might be more than this, especially if we permanently added value to the club rather than just providing a speedup. Ashley should probably answer this more.
Could you have done this alone? Could Sydney have done this alone? Should I do a residency?
Thomas thinks he would have produced ~20-40% the value had he tried this alone, due to lacking various skills that Sydney, Ashley, Brandon and others had, and generally not having much community-building experience. Thomas thinks Sydney would have produced maybe 40-65% the value; but not more due to dumb problems like âtabling is great, but the gear for tabling takes 2 people to carry and Ashley and Brandon are both busy in the mornings when itâs optimal to startâ.
If you want to do a residency, at least one of the residents should meet the requirements in the âRequirementsâ section and have university community building experience. Also strongly consider meeting with Sydney or me.
If you were to re-do the residency, what would you do differently?
Generally optimize more for the club fair. Start tabling a couple of days before club fair to test pitches. Plan a big intro event to advertise at club fair.
Spend a bit less effort on the website, or copy someone elseâs website. The main purpose of the website is a button for people to join the email list. A secondary purpose is information about programs.
Learn how to do 1-1s myself, and take some of them so Sydney would be less overworked.
If you only had, say, 100 hours, what would you do differently?
With 100 hours (say 1 resident for 2 weeks), the startup costs (getting oriented) and handoff costs are a larger proportion of the residency. To reduce these, Iâd video call my connections at the school and make sure there are 2 organizers to ensure a smooth handoff. Iâd also ask them about the culture of the school, and schedule an in-person meeting for the first day so everyone is on the same page regarding logistics, messaging, and other important things.
During the residency itself, Iâd spend as much time as possible on one-on-ones and some on tabling, and skip lower-value activities like hanging flyers and expensive messaging experiments. If emails are easy to get, I put a lot of effort into intro events, which could be well-attended despite limited tabling. If time is really short, Iâd invite organizers to a regional retreat planned by someone else rather than planning it myself, or find some other solution to fill the role of a retreat.
What do you see as the most impactful parts of a residency?
I think the first goal of a residency should be to build organizer capacity and enthusiasm to maximize the self-sustaining size of the future group. For us, this meant one-on-ones and a structured retreat. Maybe in other situations the structured retreat should be replaced with board meetings, on-campus organizer training, an unstructured camping trip, or something else.
What are the biggest challenges with residencies & how might they fail?
Pretty uncertain about this, but the handoff seems like the biggest class of problems. Succession is already a huge problem for EA groups, and succession in 2-3 weeks adds an additional challenge. Possible failure modes:
No qualified club leadership: Sometimes, despite networking, residents canât find two people who are aligned, enthusiastic, have the right skills, and willing to put in 10h/âweek each. Needless to say, this is really bad.
Club leadership resistant to residents taking over: There is an existing club, and the leadership doesnât like the residents taking over the club for several weeks. This could be because the residents and club leadership arenât aligned, or just have conflicting visions. We didnât have to deal with this because we were restarting a dormant club, but we heard this was a significant problem at other residencies.
conflicting visions: If the resident is a student from a successful uni group like Stanford/âCambridge/âwherever, the local club leadership can probably learn something about the group model. But maybe the resident is importing problems with their own group, or the Stanford/âCambridge model is inappropriate for the school. This is tricky and I see no easy solution.
Club leadership doesnât know club logistics: Someone in the club leadership should know how to use Airtable, plan meetings, interface with faculty, etc.
Club leadership has no social cohesion /â hard to form an organizing team: I think thereâs a world where we forgot to have the organizers meet each other or decide on responsibilities, so the leadership struggles to act as a team in the absence of outside guidance.
Itâs also important to think carefully about messaging. If we had been lazy and copied old marketing material that overemphasizes earning-to-give, students would get an inaccurate idea of EA. Also, every school has a different culture, and framings of EA ideas that are common at some schools can be controversial at other schools. I think itâs good to understand progressive viewpoints when at progressive schools. Residents should probably also talk about messaging with the club leadership, and with organizers at schools similar to the target school. That said, one can go too far and waste time dithering rather than tabling, and tabling is a great way to gather data on messaging.
Next target school?
Stanford EA organizers went to the Ivy League schools and MIT because they started late. But with enough capacity, especially among community builders who are not bound to the schedule of university students (recent alums, or students on a leave of absence), I think the residency model could also reach many large, highly-ranked public schools.
Notes
[1]: I almost didnât do this residency. Sydney and I were at an EA organizers retreat in Boston in August, and I heard Stanford EA organizers talking about planned residencies to various schools immediately afterwards. Sydney had decided to go to Penn, and I made the last-minute decision to join, canceling my vacation plans. We flew out to Philadelphia the next day. Sydney and I get along well (we managed to not go insane living and working together for 3 weeks), and we think this was important to the success.
[3]: I thought about the estimate for a couple of hours and am pretty uncertain about the exact number but am pretty sure the residency was good. The valuable things we did include
lowering activation energy for the Penn EA team (they have to run a club, not start one)
getting Intro Program applications and emails by tabling
transmitting best practices to organizers
being motivational/âinspiring, which transmits enthusiasm to organizers
connecting organizers to each other and other EAs in the region
These are hard to quantify, but they all seem pretty necessary for a uni group to generate value. So the question is, how fast could Penn organizers have gotten there without us? Imagine instead of doing the residency, we flew back to California and just kept remotely pestering Ashley/âBrandon to find more organizers, then flew back three weeks later for the retreat and tried to transmit as much best practices and enthusiasm as we could, with only ~25 hours of investment. I think in these worlds the Intro Program is smaller due to disorganization and lack of tabling, events have ~half the attendance, and many of the organizers donât come to the retreat and so donât get involved due to lack of one-on-ones. Eventually capacity gets built up, which takes maybe a year and a few retreats (possibly run by us). But this is already better than doing nothing.
The next marginal 25 hours are one-on-ones, which are just as good per hour because they increase the value of the retreat. The next best ~100 hours are tabling, which club leadership would do anyway if they had large blocks of time, and the next 150 hours include startup costs but also doing things like the website that the club needs eventually anyway and probably canât do in less than 150 hours. At 30 hours/âweek of club organizer time, it takes 10 weeks to do 300 hours of stuff, which is 3 months; we think we were more efficient too. Combining my vague vision of the counterfactual with this tracking of hours makes me think that 4 months is reasonable. But I should note that âsped up by 4 monthsâ has a kind of optimistic expectation that the group will keep growing in the future. So maybe a more accurate wording is âcaused a semesterâs worth of Intro Program applications, lots of momentum, and various positive intangiblesâ.
Also, itâs only been 1.5 months since we left. So the group could still die this year, which would make this post look very silly.
- [AMA] AnÂnouncÂing Open Philâs UniverÂsity Group OrÂgaÂnizer and CenÂtury Fellowships by (29 Jul 2022 18:38 UTC; 111 points)
- A model for enÂgageÂment growth in universities by (15 Dec 2021 19:11 UTC; 34 points)
- A model for enÂgageÂment growth in universities by (15 Dec 2021 19:11 UTC; 34 points)
- [AMA] AnÂnouncÂing Open Philâs UniverÂsity Group OrÂgaÂnizer and CenÂtury FelÂlowÂships [x-post] by (LessWrong; 6 Aug 2022 21:48 UTC; 14 points)
- 's comment on How to kick-start 10 uniÂverÂsity groups in 6 months by (10 Jun 2022 0:48 UTC; 10 points)
So, I wasnât really a part of this, but I think based on karma this post got people might be interested in an (unofficial) update from a random philly guy.
Iâm not aware of any evidence of Penn EA existing in 2022. My understanding is the weekly dinners got kneecapâd by omicron, organizers got busy with other projects, and it never bounced back. Iâve put together a few events using campus facilities in the meantime, but havenât done a lot to interface with the student body.
Fall 2021 may be some sort of success story, but to what end? With what metrics do we evaluate university groups with respect to time?
I think institutions take a lot of grinding, persistence, boring tasks that last way longer than any of the lessons they teach you. Intergenerational continuity is hard, what with people graduating and leaving all the time. Storytime: when I got to the community college of philadelphia, the tutoring department didnât have anyone for courses harder than calc 1. After me and my peers climbed through the sequence without tutors, we said âthat was hard; letâs make it easier on the next cohortâ and all joined the tutoring department. Then, we scouted for and trained future tutors to replace ourselves, and moved on (note, more complicated story with covid kneecapping the continuity we worked to build, so some of us came back during covid, but the lesson of the story remains). This sort of âleadership must make themselves scarce; always be exitingâ ethos is emphasized in some lefty activism Iâve done and literature Iâve read, but this comment is the first Iâm aware of it being uttered on the EA forum. I donât think I need to argue against the virtues of centralization to make a case for distributing responsibility through the group such that when an organizer feels called to another project everything doesnât just stop in itâs tracks, but some intuition pumps like âeggs in one basketâ, âsingle point of failureâ, âchokepointâ are important to keep on a post-it note somewhere in your mind, in case youâd like to act on the criticisms they imply.
Random spitballing: Isnât greek life very old? Did anyone study frats, sororities, whatever culty stuff is going on with the exclusive clubs at harvard (as portrayed in the movie the social network)? They seem like theyâve built stable institutions, and nailed intergenerational continuity!
Moving forward, Iâm proposing that everyone involved in university groups make bets about the engagement theyâll get 2, 5, and 10 semesters from now. (and remember, market manipulation and insider trading are virtuous, in this case!)
One more thing, above I wrote
and I think not only does it bear repeating, but itâs important to highlight questions like âhow many EAs can we afford to assign to CB logistics, ask them to build those skills, when we need people building different skills if we want to save the world?â, âis it in an EAâs interest to build these skills w/âr/ât opportunity cost?â, etc.
Thanks for posting this comment.
TL;DR: There is basically no Penn EA group right now. However, I donât think this is as severe a failure as it sounds because all the potential organizers might be doing higher-impact things.
Basically, the first couple months of Penn EA found ~6 engaged people with any time at all to help run a group, not including me and Sydney. My impression is that one or two of them drifted away from EA as a philosophy, stopped having time, or something. Three of them dropped out of Penn (undergrad or graduate); of these, Ashley and Akash are working on movement-scale talent search and Tamera is skilling up towards direct work. They thought about their decisions pretty carefully from an impact maximization perspective, and I think they are creating substantially more impact than they could have at Penn. The other potential organizer, Brandon, is a full-time student who was pretty new to EA and didnât have capacity to run a group on his own.
My best guess is that under the optimal allocation of people, 0-1 of the 3 who dropped out are still at Penn doing community-building, and 2-3 of them correctly dropped out.
Brain drain towards direct work and bigger meta work is an established pattern with university groups. The Stanford EA executive board had 9 people in early 2021, and for a while in late 2021 Stanford EA was basically being run by 1-2 people and in danger of falling apart because basically the entire board graduated, dropped out, or spent all of their spare time doing part-time EA things that were not running Stanford EA. I think that most of them made good decisions. To be fair, a little of this was for bad reasons: when everyone else is moving to Berkeley/âthe UK, itâs fun to be there and so on. Also, even if everyone had absolute disadvantage running Stanford EA, there should be coordination among people so that the person with the least comparative disadvantage among Stanford EA made succession not fail. (Although I know a lot of Stanford EA people, some of this is rumor/âspeculation so donât take my word as definitive.)
This was all exacerbated by three shifts in movement-building philosophy over the last few months.
The first is from âEA group community-buildingâ to âglobal talent searchâ. There is now less emphasis on having a university group at every top-20 university in the US, and more emphasis on (a) search for top talent in neglected countries like India and Brazil (either at top universities there or through other means), and (b) cause-area-specific discussion groups at the very top universities like MIT. Due to all the longtermist resources now, this is happening the most with AI safety reading groups, some following Cambridgeâs AGISF syllabus, some more advanced that are filled with aspiring researchers. But itâs also happening with e.g. altpro. I think this shift is on net good but depends heavily on implementationâthere can be old-style groups that cause a lot of impact on important problems because they happen to target talent well and are large enough to specialize into cause areas, or whatever reason.
The second shift is from intro/âadvanced fellowships to retreats/âworkshops. Workshops have much higher fidelity, plus concentrate people from multiple universities in a space so they can have high-context conversations and connect with each other. Thereâs a tradeoff here, but my view on that is outside the scope of this comment. I think itâs potentially OK that Penn didnât have fellowships for this reason, though itâs pretty bad that they didnât have weekly dinners.
The third shift is towards greater ambition and larger action spaces. This is why most of the organizers dropped out. I think Penn had a greater proportion of people who were willing to drop out than most university groups, and this has diminishing returns when it is as high as 75%.
Lessons for new university groups
If thereâs an update for new university groups, itâs that building capacity to recruit talented people at moderate efficiency is easy, but that succession fails by default and is even more likely to fail when you donât put lots of effort into it. The community is moving towards a place where it might no longer be necessary to developing university groups in the same way, and succession is somewhat less of a perennial issue, due to workshops and efforts like GCP. However, thereâs value in university groups existing at all, e.g. by creating ways for people to hear about EA at all. Thereâs also a lot of value in building groups with their own culture (this gives you intellectual diversity and information value) though this is easy to screw up either by reducing efficiency or targeting the wrong things.
The top considerations I have for whether to do community-building at university groups, for people who are willing to drop out, are basically âwhatâs your counterfactual? (if movement-scale CB, does this have higher fidelity and growth rate?)â, and âare you irreplaceable in this university CB role?â and âif you build the university group, will it actually produce a lot of impact (the classic source of impact is career changes of highly talented people towards the best thing they could be doing)?â. There are sub-considerations which I can expand on, but I havenât spent much time thinking about university group strategy, so Iâm pretty uncertain.
Mistakes
I think the mistake was that Sydney, Ashley, and I basically stopped thinking about Penn when we left. We all have pretty limited capacity to take on side projects, but one of us should have (a) tried to have weekly calls with the remaining Penn organizers, and (b) connected them with GCP. I think the median outcome of this is that not much happened at Penn due to sheer lack of organizer hours, but there was still lots of expected value there for relatively little investment.
I see two open discussions here.
One is brain drain, as you mentioned. I wrote a little about this in the âkeeping oneâs eye on the ballâ section of this comment. I think we should be reasoning about the bay as âlook at all these magical things happening thereâ and constantly panicking about the opportunity cost every day weâre not creating more bays in other places.
Another has to do with
And why Quinn M wasnât tapped, or myself. Is there a view formed that people in their later 20s out working arenât a super good fit for university work? Is there a view formed that top universities are culturally particular, and that people who werenât at top universities would screw it up? Things like this seem plausible to me, but Iâm shooting in the dark.
But moreover, Iâm really glad to read your comment about weighing CB against anything else CBers could be doing. (I have concerns about the movement doing so much advocacy that we build out the wrong skillsets, and so on).
My view is some of (1) and not much of (2), and people who think more about university groups might have more concerns. The point at which we made the mistake was probably not even thinking about it, and itâs plausible that if we had, we would have connected them to EA Philly.
Iâve spent a couple hours reflecting on this and think the Penn residency was quite successful. As an organizer, I am incredibly grateful to have been able to shadow Sydney and Thomas for a couple weeks and help out /â learn how to do things like 1-on-1s, run weekly dinners, set up club logistics, etc, (I was probably spending 20+ hours/âweek supporting the residency).
That said, Iâm wary of people updating too much in favor of residencies based on this post.
Stanford EA ran a couple of residencies at other top universities on the East Coast during this time frame. Most of these other experiments didnât seem to produce as great of results. Perhaps this was because other schools received a lot less organizer time (for example, Princeton received one FTE organizer for a couple of days, whereas Penn had two FTE organizers for three weeks). However, I also think an important consideration is that Sydney knew me (Ashley Lin) prior to coming to Penn. As Thomas mentioned, I am on a gap year and had ~20 hours/âweek to jump in and help out however necessary.
The residency seems a lot harder if you donât already know a potential organizer at the school and if that person doesnât have much time (is too busy at the start of the school year). To me, the process for making a uni group self-sustaining looks something like:
Identify potential organizers, probably through 1-on-1s if you donât know people already
Screen them for alignment and general competence, probably through working with them for a couple weeks during the residency period
Get a commitment from that person to take responsibility for making sure the group doesnât die, before handing over the reins
It seems like if you didnât already know someone, Phase 1 would take up a significant amount of energy (and you might not even find a suitable candidate). This might not leave enough time for Phase 2 to happen during the residency period, making Phase 3 an automatic failure point. My crux for whether a residency should happen: one should already have identified a promising organizer who would want to shadow the experienced organizers during the residency period.
Instead of seeing the value of residencies as something that speeds up a group by ~4 months, I think one should see it as a training phase for promising organizers. Iâd even be excited by experiments where experienced group organizers choose a school to do a residency at, and we brought promising organizers from other schools to observe and help out. As someone who did this âtraining phaseâ and now feels comfortable helping start EA groups at any uni, I think mentorship during residencies could be one of the most effective ways to help students upskill in group organizing /â meta-EA work.
Iâm also quite uncertain about the ~4 month speedup, but my instinct is that if Sydney and Thomas didnât do the residency, Penn EA would look like what it does now a semester later.
I think the primary value that the residencies provided are points 4 and 5, âbeing motivational/âinspiring, which transmits enthusiasm to organizersâ and âconnecting organizers to each other and other EAs in the region.â Lowering the activation energy is great, but if you get competent organizers excited enough, it actually isnât too much of a unique value. Tabling is great, but I think there were diminishing returns and the 80â20 would have been just tabling at all of the club fairs and maybe a couple days on Locustâwhich seems pretty do-able if organizers are excited enough. Essentially, I think âdoing thingsâ is generally replaceable and the unique value of residencies is probably getting organizers excited and connected.
Without Sydney and Thomas, I think it wouldâve taken me about a semester to get sufficiently âexcited and connectedâ to become a good organizer. I think this wouldâve come from a combination of attending many of the uni group organizer retreats in September/âOctober (although itâs unclear how many of these I wouldâve attended if I didnât know Sydney/âKuhan that wellâdonât think I wouldâve been invited otherwise) and being at EAG. I now feel like I have a sufficient number of high quality connections in EA (people I feel comfortable reaching out to for a favor) and have the resources I need to make Penn EA really great if I put my mind to it. Iâve noticed that many of the other organizers at Penn EA seem to be orders of magnitude more excited about organizing after attending retreats or EAG (ex, one current Penn organizer applied for a CBG after attending the third retreat), which makes me place more weight on this.
I can believe this. If so, then the most important quality in a resident might be a motivational/âinspiring temperament or something. If this is a rare property that Sydney or I have, it might prevent most residencies from being as good as Penn. But even when doing things is replaceable, Iâd still be excited by residencies that pour more time into organizer groups at the start of the year at an efficiency level roughly similar to the marginal organizer hour, especially when the school is large enough that returns diminish slowly.
Nitpick, but I donât think returns were diminishing, unless you think there are diminishing returns to more emails gathered on overall club quality, because we maintained similar emails/âhour numbers for most of the 2 weeks we were tabling.
Where do the new organizers come from? Are they people who are already quite familiar with EA, and if so, why werenât they already involved with organizing Penn EA?
I heard EA students from Stanford EA (and maybe other EA groups too) were also helping with Princeton Effective Altruism at the start of the semester, assisting with activities such as tabling. How successful was that?