ScotGrid - Recent changes [en]

ScotGrid - Recent changes [en] 2013-05-11T22:37:03Z Track the most recent changes to the wiki on this page. MediaWiki 1.4.4 Tech Meeting Minutes 20130805 2013-05-08T11:09:41Z 2013-05-08T11:09:41 2013-05-08T11:09:41Z

New page<div>ScotGrid Technical Meeting 7 May 2013 - Present: Mark M (chair) Mike J Ewan S Sam S (minutes) Stuart P Dave C Gareth R Andy W - Durham Mike: not much to report. Unexplained aircon shutdown monday evening. Cause not identified, but did not retrip after reset. Working on SL6 - have a reasonable base install ready for building middleware services on. Start with WN? Ewan made good progress on nagios/puppet monitoring. Everything still work in progress. Have ticketed biomed regarding excessive number of jobs. Tried banning a "bad" biomed user, but this seemed to cause problems with the CE for other users. (The jobs wouldn't complete before new jobs came in, in batches of 1000). Glasgow also has the same problem, with sets of 4000 jobs at a time. There appears to be one problematic user (for both Glasgow and Durham). For Durham, the problematic user is Romain Reuillon. This is also the problematic user for Glasgow. - ECDF Andy: nothing exciting. SL6 move is still ongoing. Working SL6 CE, seems to work (can't put into test due to pending config needed on SGE end). Still have network restrictions all across SL6 cluster - meeting last week, trying to avoid hacky solutions. Need to push to remove ACL restrictions. Squid deployment request (for upgraded WLCG monitoring). Did anyone else do it? (Glasgow has not.) Andy will try to make the relevant change today at ECDF. - Glasgow Mark: pile production reduced from ATLAS thanks to changing max mem limit for production to 3GB. This means we now run at much better efficiency. Compressor issues with 243d caused a problem, meaning we had to reduce capacity. David: returning from holiday and reacclimatising. Graphite: will send around details on our config, but after some refactoring has been done to make it more efficient. Sam: test DPM (IPv6 reverse lookup doesn't work, so you have to add /etc/hosts entries) on SL6 IPv6, with all transfer protocols enabled. Stuart: WMS L&Bs all updated to EMI3. Just need to do the ARGUS integration. The principal users are pheno, so hopefully Durham didn't notice too much disruption. Gareth: compressor diagnostics - runs very hot. Could be compressor, could be blockages. - AOB F2F meeting in Durham - fill in Doodle poll please! Waiting on CHEP acceptance for presentations. EGI Tech Forum in Madrid 16th September. Call for papers expected soon. GridPP31 is 24-25th September. Imperial College. -- Chat Log 11:00:09] Ewan Steele joined [11:00:15] Mark Mitchell joined [11:01:39] Mike Johnson joined [11:02:02] Mark Mitchell Can hear you fine [11:03:34] Gareth Roy joined [11:03:48] Andrew Washbrook joined [11:03:53] David Crooks joined [11:05:56] Stuart Purdie If it was 'hot' in the local area when the Air Con tripped out, might be due to the hot side overheating? We've seen that up here once or twice, but it can be difficult to pin down with enough precision to get people to do stuff about it. [11:08:35] Gareth Roy 4826 biomed [11:08:38] Gareth Roy at the moment [11:08:54] David Crooks left [11:09:38] Mike Johnson Ours is Romain Reuillon. I'm considering a trip to France to break some legs [11:10:39] Gareth Roy My bad [11:10:40] Gareth Roy %2fo%3dgrid%2dfr%2fc%3dfr%2fo%3discpif%2fcn%3dromain%20reuillon:biomed [11:10:52] Gareth Roy svr012:~# qstat |grep biomed047|wc3942 23652 342954 [11:12:24] David Crooks joined [11:23:17] Mike Johnson http://www.doodle.com/g2iyzenrny2wh9gx </div>

Mark mitchell foobar Tech Meeting Minutes 20132404 2013-05-08T11:08:00Z 2013-05-08T11:08:00 2013-05-08T11:08:00Z

New page<div>ScotGrid Face to Face at ECDF</div>

Mark mitchell foobar Tech Meeting Minutes 20130105 2013-05-08T11:06:56Z 2013-05-08T11:06:56 2013-05-08T11:06:56Z

New page<div>ScotGrid Technical Meeting 1 May 2013 Present Sam S (chair, minutes) Gareth R Ewan S Mike J Wahid B Andrew W - Durham Mike: not much going on. Have had issues with biomed queueing thousands of jobs for few cores (which did make the batch system very upset). Tried banning the worst offender, but the ban seemed to cause seg faults in the CREAM CE logs. (This is also a problem we've seen at Glasgow, regarding excessive biomed jobs) Suggested that Mike should GGUS ticket Biomed for excessive resource use before removing biomed support entirely. DPM daemon stopped again, apparently might be triggered by backup dump process? (Broke ATLAS Production, but resolved when daemon brought up again.) [Check that mysql backend is set up appropriately for lockless dumps - binary logs, etc, single-transaction for the dump.] 10Gig NICs going into disk pool nodes. Waiting on certificate. Pressing on with SL6. Have a test SL6 UI, but having a few teething problems. Ewan is going on with his nagios-controlled-by-puppet project. Puppet configures nagios from a database of nodes. Site has been relatively stable, other than some UPS blips caused by the site being at the peak power load for infrastructure. Also tweaked config to make the 12 core nodes actually have 12 job slots. (They'd been getting the "default" 8 core config.) ECDF Andy: nothing much to report. SL6 deployment inches along. Meeting with the systems team to talk through current network traffic issues. Short term: can avoid restrictions centrally, using some VLANing etc. SL6 CE now visible externally - waiting on a new host cert for it, before we can start tests. In parallel, have been trying to get WNs up to speed for SL6 compat testing… LHCb problem, "gone away" - upgraded the WN TARBALL which magically fixed things. LHCb not interested in pressing this problem further. Xrootd analysis queues. Wahid: on Friday tried to turn on direct-IO for Analysis using xrootd. Seemed to break some user jobs, but not very well understood what problem was (little feedback on list). Sched-config problem is that the entire pilot config is labyrinthine and poorly documented and can sometimes be site dependant. (TAIWAN are apparently successfully doing this?) Glasgow Gareth: Very grumpy with ATLAS. Might understand issue with pile jobs: that they are 4GB (3800GB) jobs. With 2GB swap per process, on a 32 to 64 GB machine, there is a physical limit on the swapping process (copying all memory to disk takes 8 minutes just as a physical stream). As a result of some discussions with ATLAS, we've turned our max ram per job down to 3GB, and our job efficiencies are now back to normal levels (so far), despite running pile. Both WMSen are offline due to still being on EMI-1. Problems with EMI-3 L&B have held back their upgrade (we might need to tear them down rather than upgrading in place). Sam: DPM fell over due to 64000 open connections on the DPM xroot federator port. Not sure what the problem was, services came up again once xrootd was restarted. AOB - ScotGrid F2F 2013 Durham edition: July might be a bit close (and also rules out attendance by management types). Mike will consider if August 1st or 3rd weeks are doable. Doodle poll will be circulated. - Ewan wanted to create a custom fact in puppet for factor. The room was lacking the knowledge on doing this (so we're interested to know if it works. - Lydia Heck & PRACE Could she just be given access to the Durham UI to give her a "free" gridftp service? (It sounds like she's already installed some of the globus toolkit from EPEL.) - 11:04:32] Gareth Roy So looks like Mark forgot to book a meeting [11:04:48] Wahid Bhimji hi [11:05:31] Mike Johnson fightig with evo as usual [11:06:05] Andrew Washbrook joined [11:06:24] Gareth Roy Yeah we've seen that... well only 4K here [11:06:47] Ewan Steele joined [11:09:25] Wahid Bhimji we (ecdf) removed biomed a while ago - couldn't be arsed with it. [11:09:54] Ewan Steele left [11:10:13] Ewan Steele joined [11:12:49] Wahid Bhimji how do you do the dump / backup [11:13:16] Gareth Roy Easy way to check [11:13:17] Wahid Bhimji single-transaction [11:13:22] Gareth Roy move the backup time by a few hours [11:13:25] Gareth Roy see if it fails [11:21:58] Andrew Washbrook ! rant ahead ! [11:26:35] Andrew Washbrook I am on shift today - do you want me to reopen the ATLAS ticket? [11:28:13] Mike Johnson Romain Reuillon is the guy we banned. Then unbanned. [11:38:09] Wahid Bhimji left [11:38:13] Ewan Steele left [11:38:15] Andrew Washbrook left </div>

Mark mitchell foobar Tech Meeting Minutes 20131704 2013-05-08T11:05:01Z 2013-05-08T11:05:01 2013-05-08T11:05:01Z

New page<div>ScotGrid Technical Meeting 17 April 2013 Present: Sam S (minutes) Andy W David C Gareth R Mark M (chair) Mike Johnson Ewan Steele - Edinburgh: Andy: LHCb issue (still rumbling on). LHCb pilots arrive, run, die on worker nodes (environment setup fails). Process with LHCb is a bit strange - hard to progress with Raja or Vladimir, since they're both too busy. Possibly a problem with the version of the Worker node tarball, but only LHCb seem to be affected. New release of WN-TAR will be installed in due course. Trying to deploy new services moving to SL6. Various problems using the new equipment which is on a new subnet (which is managed by the network team, not the ECDF team). Communication issues between the two are slowing things (NFS traffic seems to not work between the subnets). - Durham: Congratulations to Ewan Steele on a baby girl. Glasgow: Mark: Some issues with cooling: had to have compressors for one AHU in 243 cleaned out, since they were failing, and there are issues with the alarms on the chillers in 141 (which need sensors replaced). Working on reduced capacity while these fixed. Have been given Money for infrastructure improvements to 243/141, reliable cooling + power, including generator. Dave: looking at monitoring. (recap: refreshing monitoring using graphite as the engine and central data collection point.) Mostly recasting after meetings and being annoyed by (apparently spurious) tickets against our ARGUS server (which is EMI2 even if the monitoring says it's EMI1). We are having intermittent problems with the WMSen at the moment - this is complicated by the fact that we also have to move to EMI3 (preferably) on them. Stuart is covering this, though. Sam: EGI community forum. Par. Gareth: EGI Community forum. Cavorting. Air con stuff in 243. Puppet on the test cluster. Gareth asked about other groups experience in storing config data for puppet outside of puppet - zookeeper, etc Mike: we are starting to look at using Puppet to manage nagios install. Will be keeping data about how nodes are setup, probably in ldap or sql or some other data source. Ewan: our general plan is to store the config information and use it to automatically generate the nagios config. Apparently well documented on the internet. Gareth & Ewan will share notes & links on stuff. Mark: we're considering decommissioning the WMSen, as they're a bit of an albatross. - Durham Mike: Two tickets outstanding: 1) LHCb user mapping 2) EMI DPM thing. Briefly got put into test in hammercloud, and Mike rebooted the SE (which caused some concomitant problems, as that SE sometimes forgets its hostname). 20TB of disk is in the cluster, so we can talk to Sam about configuring that for DPM (using puppet). The disk filesystem + etc are installed, so we just need to add the filesystems to DPM. We also have a test EMI3 UI. Had problems upgrading directly from EMI2 -> EMI3 (java package dependancy issues). *see discussion in chat window. Waiting on some 10GigE cards for the new new disk servers, so they're waiting 'til then. AOB: Scotgrid F2F next week. Andy to follow up on precisely what's going on for this as Pete Clarke has been a little unresponsive. Hopefully meeting will start at around 10 to 10:30. 10:57:11] David Crooks joined [10:57:40] Gareth Roy joined [11:02:13] Andrew Washbrook joined [11:03:14] Mark Mitchell joined [11:12:14] Ewan Steele joined [11:12:34] Mike Johnson joined [11:15:17] Mark Mitchell Is your audio working Durham [11:15:44] Sam Skipsey Come in Durham, Glasgow calling! [11:16:08] Sam Skipsey (And the votes from Durham: [11:16:13] Sam Skipsey ECDF nul point! [11:16:21] Sam Skipsey Glasgow 1 point!) [11:16:41] Gareth Roy Yay!! Congratulations don't listen to Mark [11:17:49] Mike Johnson left [11:18:05] Mike Johnson joined [11:19:16] Andrew Washbrook cash? [11:19:32] Gareth Roy We need power more than compute [11:19:37] Gareth Roy [11:20:21] Sam Skipsey The funding is entirely in oddly shaped pieces of gold. [11:23:58] Gareth Roy What?!?!?! [11:24:17] Andrew Washbrook Sam is speaking in tongues [11:27:00] Mike Johnson left [11:27:15] Mike Johnson joined [11:32:49] Gareth Roy Noise form next door [11:36:06] Mark Mitchell from or form? [11:36:30] Gareth Roy no comment [11:38:28] David Crooks We had a similar problem with gLite3.2 -> EMI 2 [11:38:32] Gareth Roy https://www.gridpp.ac.uk/wiki/Staged_rollout_emi3 [11:38:36] Gareth Roy Wiki Link [11:38:36] David Crooks Because of orphaned packages [11:38:47] David Crooks Yep [11:41:25] David Crooks 3 more sleeps! [11:45:15] Andrew Washbrook left [11:45:16] Gareth Roy left [11:45:17] Ewan Steele left [11:45:18] David Crooks left [11:45:22] Mark Mitchell left [11:46:05] David Crooks joined [11:50:18] Mike Johnson left </div>

Mark mitchell foobar ScotGrid Technical Meetings 2013-05-08T11:03:18Z 2013-05-08T11:03:18 2013-05-08T11:03:18Z

/* 2013 */ <table border='0' width='98%' cellpadding='0' cellspacing='4' class='diff'> <tr> <td colspan='2' width='50%' align='center' class='diff-otitle'>Revision as of 11:03, 8 May 2013</td> <td colspan='2' width='50%' align='center' class='diff-ntitle'>Current revision</td> </tr> <tr><td colspan="2" align="left">Line 19:</td> <td colspan="2" align="left">Line 19:</td></tr> <tr><td> </td><td class='diff-context'>* [[Tech Meeting Minutes 20132404]]</td><td> </td><td class='diff-context'>* [[Tech Meeting Minutes 20132404]]</td></tr> <tr><td> </td><td class='diff-context'>* [[Tech Meeting Minutes 20130105]]</td><td> </td><td class='diff-context'>* [[Tech Meeting Minutes 20130105]]</td></tr> <tr><td colspan="2"> </td><td>+</td><td class='diff-addedline'>* [[Tech Meeting Minutes 20130805]]</td></tr> <tr><td> </td><td class='diff-context'></td><td> </td><td class='diff-context'></td></tr> <tr><td> </td><td class='diff-context'>===2012===</td><td> </td><td class='diff-context'>===2012===</td></tr> </table>

Mark mitchell foobar