From: Jim Brown (Jim.Brown@xxxxxxxxxxxxx)
Date: Tue Oct 02 2001 - 14:55:47 GMT-3
My point is, I cannot see how exams could be graded without some sort of
review by a human... ever. I'm know they use utilities to check for some
basic functionality and determine what should be reviewed by a human. If you
can't ping it, it probably isn't configured correctly!
On my statement regarding the intricacies of auto grading without very
specific design requirements. I could solve a route manipulation problem
with a distribute list, as-path list, route-map using a distribute list, or
a route-map using an as-path filter list, a distribute list into the process
(I'm not sure on the last one). I could attack it from the neighbor
statement out, neighbor in, at the interface out, at the interface in. There
are tons of ways to solve the problem and the requirements usually leave
some room for interpretation.
I also left out extended access lists or prefix lists which could include
multiple lines or a single statement depending on laziness or time to arrive
at the correct filtering result. Suppress-map with the appropriate permit
and denies....
It would be difficult at best to design a program that would take into
account every possible permutation and produce an accurate grade without
extremely specific upfront design requirements. Specific detailed
requirements would remove some of the deep understanding of the technologies
necessary to pass the exam.
My opinion is a pair of human eyes will always be necessary to properly
grade configurations from a properly difficult lab examination.
And I think you originally used the term "automate grading" in your post.
-----Original Message-----
From: Brian Dennis [mailto:brian@5g.net]
Sent: Tuesday, October 02, 2001 10:15 AM
To: Jim Brown; ccielab@groupstudy.com
Subject: OT: RE: New lab format starting tomorrow!
Jim,
What are you talking about? I don't think anyone used the phrase "complete
auto-grade" but I personally think that using a script (software) to grade
most of the exam is a very useful tool.
Do you really think that software couldn't detect if you used a static
route? Anyways here is part of an e-mail from a proctor at Cisco.
******************
The rumors you have heard refer to a tool which we (CCIE Program) have
been developing for about 2 years. It's called E-Proctor. This tool doesn't
replace the proctor but is simply a grading tool that assists in grading
certain exams that it has been configured to handle. This tool is hopefully
going to allow us to handle the grading of more candidates per proctor and
thus increase our capacity without the need to hire more proctors.
******************
Also where did you come up with this?
"I think any proctor would have your head at the suggestion the labs are
auto graded and they just hang around all day to answer questions."
Brian Dennis, CCIE #2210 (R&S)(ISP/Dial) CCSI #98640
5G Networks, Inc.
brian@5g.net
> -----Original Message-----
> From: Jim Brown [mailto:Jim.Brown@CaseLogic.com]
> Sent: Tuesday, October 02, 2001 8:52 AM
> To: 'Brian Dennis'; Jim Brown; ccielab@groupstudy.com
> Subject: RE: New lab format starting tomorrow!
>
>
> As far as I understood, the auto grading of the test only does
> some sort of
> a recon ping scan. Can I actually ping this interface? If not, the
> requirement is not working and no further inspection is
> necessary, if I can
> ping then I should inspect and make sure the features were implemented in
> the correct fashion. ALL BASED ON CONJECTURE.
>
> Heck, If it is auto graded as you suggest, static routes here I
> come. I know
> for a fact the proctors examine the configs to look for certain items. I
> have watched them grade configurations.
>
> A partial auto grade weeds out unnecessary review of non-working configs.
>
> I think any proctor would have your head at the suggestion the
> labs are auto
> graded and they just hang around all day to answer questions.
>
> For complete auto grading the exams would have to be so specific about how
> to implement a solution they would almost need to tell you how to
> implement
> it.
>
> -----Original Message-----
> From: Brian Dennis [mailto:brian@5g.net]
> Sent: Monday, October 01, 2001 6:25 PM
> To: Jim Brown; ccielab@groupstudy.com
> Subject: RE: New lab format starting tomorrow!
>
>
> Jim,
> The lab is already being graded by software. Ask a proctor for yourself.
> Cisco's in-house software is called eProctor. There is still some human
> review of the results but the vast majority of the labs are graded by
> software.
>
> I did a contract with Cisco in '98 as a Software Tools Automation Engineer
> where I developed scripts to automate testing of various IOS
> features. I can
> assure you that it's not impossible to automate the grading. For example,
> why would I need to know what you named your route-map? All I
> have to do is
> find the route-map that is applied to the redistribution command or
> interface to determine what name you used. As far as prefix-list or
> access-list it would be easy to put a condition in to check for
> either one.
>
> Brian Dennis, CCIE #2210 (R&S)(ISP/Dial) CCSI #98640
> 5G Networks, Inc.
> brian@5g.net
>
> > -----Original Message-----
> > From: nobody@groupstudy.com [mailto:nobody@groupstudy.com]On Behalf Of
> > Jim Brown
> > Sent: Monday, October 01, 2001 4:36 PM
> > To: 'Libone Mhalanga'; ccielab@groupstudy.com
> > Subject: RE: New lab format starting tomorrow!
> >
> >
> > I cannot see how the exam could ever be auto-graded.
> >
> > How could they know what you would name your route-maps or
> > whether you would
> > use prefix/extended access-lists?
> >
> > The permutations are limitless and your work could only be verified via
> > trained eyes.
> >
> > For auto-grading to even be considered, test requirements would need to
> > specify names for route-maps, prefix-lists, access-lists, and
> the like. At
> > that point a monkey could configure the scenarios if told which
> > commands and
> > technologies to use.
> >
> > Someone made a funny comment regarding the extremely specific design
> > requirements necessary for auto-grading...... We could all become CCCAPE
> > (Cisco Certified Cut And Paste Experts).
> >
> > I would like to purchase one of those San Jose lobby chairs after I
> > successfully complete the lab, just to burn it (you might be able
> > to relate
> > if you have taken the lab in San Jose). I'm starting to hate those damn
> > chairs.
> >
> > Have some sympathy for the proctors. They will now have to deal
> with twice
> > as many exams to grade, although no arguing, hostility, or
> > removal of upset
> > candidates during the debriefing.
> >
> >
> >
> > -----Original Message-----
> > From: Libone Mhalanga [mailto:libone@digisle.net]
> > Sent: Monday, October 01, 2001 3:16 PM
> > To: ccielab@groupstudy.com
> > Subject: RE: New lab format starting tomorrow!
> >
> >
> > Well in my opinion anything that removes human intervention in deciding
> > the outcome of this exam is most welcome !!
> > Proctors are NOT above human failings and prejudices. I certainly found
> > that in Brussels two weeks ago !!
> >
> > -----Original Message-----
> > From: Bruce Williams
> > Sent: Mon 10/1/2001 1:17 PM
> > To: John Kaberna; ccielab@groupstudy.com
> > Cc:
> > Subject: Re: New lab format starting tomorrow!
> >
> >
> >
> > I hope that it does not come to that. I have heard people
> > express their
> > concerns about the CCIE losing it's value and I always thought
> > these
> > concerns were invalid, but I believe that this is something that
> > will
> > devalue the CCIE. If the CCIE goes to Sylvan testing, it will be
> > easier for
> > people to cheat and you will see a dramatic increase in the pass
> > rate.
> >
> > Bruce
> >
> > ----- Original Message -----
> > From: "John Kaberna" <jkaberna@netcginc.com>
> > To: <ccielab@groupstudy.com>
> > Sent: Monday, October 01, 2001 3:56 PM
> > Subject: Re: New lab format starting tomorrow!
> >
> >
> > > Computer based marking is not next. However, Cisco already
> > has plans in
> > the
> > > work to extend testing to other remote Cisco sites and have
> > the proctor be
> > > available over a webcam. The next step is Sylvan testing. I
> > got this
> > info
> > > first hand from a proctor. It's not a rumor. Whether it
> > happens or not I
> > > think is still in discussion.
> > >
> > > John Kaberna
> > > CCIE #7146
> > > NETCG Inc.
> > > Cisco Premier Partner
> > > www.netcginc.com
> > > (415) 750-3800
> > >
This archive was generated by hypermail 2.1.4 : Thu Jun 20 2002 - 22:33:11 GMT-3